rds_exporter breaks when some instances don't have enhanced monitoring enabled

Description

Impact on the user: Addin only one RDS instance w/o  enhanced monitoring is stopping PMM from collecting  OS-level data for all instances. No data for  RDS  with enhanced monitoring on  

 

There are issues when mixing instances with and without enhanced monitoring to PMM. After adding any instance without enhanced monitoring, the OS-level metrics stop being collected for ALL existing instances. 

Steps to reproduce:

  1. Add RDS/Aurora MySQL instance (instance1) configured with enhanced monitoring=on

  2. Check Node Summary for instance1, all good

  3. Add another RDS/Aurora MySQL instance (instance2) with enhanced monitoring=off

Actual result:

Check Node Summary for instance1, no metrics reported anymore

Expected  result:

Node metrics are still present for instances where  enhanced metrics ON 

Workaround: 

Turn on Enhanced monitoring to all RDS instances. Note: this will cost $, so not a good workaround if the user needs to save some money  

 

How to test

None

How to document

None

Attachments

12

Activity

Show:

Ivan Groenewold 
December 22, 2021 at 12:31 PM

hmm... the only thing I can think of that seems to be different from my test case is I was using IAM role instead of access keys. Can you try that?

Jiří Čtvrtka 
December 22, 2021 at 12:27 PM
(edited)

 Checked with auroras. Take a look if I didnt miss something. Thank you.

Ivan Groenewold 
December 21, 2021 at 11:43 AM

Hi, I see you are testing with Postgres instances. Can you try reproducing by following the steps as listed in the description but with MySQL instances? Also take a look at the related ticket where we fixed this for PMM 1.x I am sure the issue is the same.

Jiří Čtvrtka 
December 21, 2021 at 11:36 AM

 For instance with enhanced metrics it is ok. Disable one is not even in dropdown. So this is correct right?

Roma Novikov 
December 21, 2021 at 8:33 AM

I recommend checking "do we have metrics" not on the DB dashboard but in VM or metrics themselves  (ex: Advanced Data exploration ) because DB dashboards are designed to extract data from different available sources.
Our  RDS exporter exposes several metrics, some of which are about the same part of the system.  

Enhanced monitoring is more preferable way as it's consumes less  API calls the basis one (expectations)

Details

Assignee

Reporter

Priority

Components

Needs QA

Needs Doc

Story Points

Affects versions

Created October 27, 2021 at 12:36 PM
Updated November 13, 2024 at 9:46 AM