Description

When you add an RDS instance to PMM (https://docs.percona.com/percona-monitoring-and-management/setting-up/client/aws.html#adding-an-amazon-rds-aurora-or-remote-instance), PMM will keep track of which aws_access_key and aws_secret_key was used by each instance (as you might have different AWS users to monitor different RDSs). It uses that aws_access_key/aws_secret_key combination to retrieve OS data from CloudWatch.

If you rotate the key (generate a new aws_access_key/aws_secrete_key and disable the old credentials), PMM stops tracking OS metrics*, as it will fail to authenticate to CloudWatch with the old credentials (as expected). 

Currently, the only way to update each instance's aws_access_key and aws_secret_key is by removing the instance and discovering it again with the new credentials, which is not doable for a large number of monitored instances. 

One workaround is to manually update the aws_access_key and aws_secret_key in the PMM Server's PostgreSQL database (it keeps the information on the database pmm-managed, table agents).

It would be nice to have a way to do this through the PMM GUI, as rotating keys regularly is a security best practice, and currently, it is not an easy task. 

 

*PMM only seems to stop tracking the OS metrics once you discover a new RDS instance, as it seems to refresh a token when that is done. Here is the output from the RDS_EXPORTER logs:

 

How to test

None

How to document

None

AFFECTED CS IDs

CS0038570

Activity

Show:

Leonardo Bacchi Fernandes August 23, 2023 at 9:19 PM

Hello Roma, 

Yes, that is the long-term solution the customer is going with, and it avoids this issue altogether. If I'm not mistaken, it doesn't work if the PMM account is not in the same AWS account as all the monitored RDSs, but it is the best solution otherwise.

 

Roma Novikov August 23, 2023 at 9:06 AM

As a good workaround here - use AMI roles for the PMM server and rotate them instead of credentials.

https://docs.percona.com/percona-monitoring-and-management/setting-up/client/aws.html#creating-an-access-key-for-an-iam-user

Pinned fields
Click on the next to a field label to start pinning.
Details

Assignee

Unassigned

Reporter

Leonardo Bacchi Fernandes

Priority

Needs QA

Yes

Needs Doc

Yes
Smart Checklist
Created August 18, 2023 at 9:40 PM
Updated July 23, 2024 at 1:08 AM