Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

If 1 RDS instance doesn't have logs or enhanced monitoring turned on all RDS collection fails

Description

I recently started working on these existing MySQL RDS instances, which don't have Enhanced monitoring turned on, some have no logs going to file or Cloud Watch. (yeah I know)

Unfortunately adding any of these instances to PMM causes ALL the RDS CloudWatch metrics collection to fail. 

  • All the graphs at the bottom of the MySQL Overview show nothing

  • Cloud Amazon RDS / Aurora MySQL Metrics Grafana page displays a red message with  Templating init failed - Metric request error

  • The pmm-rds_exporter-9042.service crashes and restarts endlessly

I have confirmed this by using a test instance of PMM and adding to it just 1 host which I know has Enhanced monitoring on & logs enabled.  With that setup I get graphs at the bottom - but I still get the Templating init failed.

This should be more robust and collect the metrics that it can. CloudWatch (non-enhanced) is showing metrics in the AWS console, so there is a lot which can be pulled down anyway.

 

  1.  

  1.  

 

 

How to test

None

How to document

None

AFFECTED CS IDs

273107

Attachments

1

Smart Checklist

Activity

Lalit Choudhary February 16, 2021 at 1:02 PM

Lalit Choudhary December 22, 2020 at 7:05 AM

Please open the new report for the 2.12 version issue with bug details.

you can mention this bug in reference. 

M December 21, 2020 at 3:54 PM

this is still occurring on PMM 2.12

Lalit Choudhary December 27, 2019 at 9:56 AM
Edited

Tested with 1.17.3-Dev ,

added 2 rds instances 1 with enhanced monitoring enabled and another one with enhanced monitoring Disable.

Restarted pmm-server

  1. I do not see a panic error in rds_exporter logs

  2. rds_exporter running fine

  3. it has error since one of the rds instance not enabled with monitoring. 

  4. And for rds instance with monitoring enabled we can see all graphs 

The only problem here for keeping rds instance which has enhanced monitoring Disabled is, it will keep adding error message for  "component=enhanced source="scraper.go" every 10 seconds. 

Removing such RDS instance from monitoring will fix this issue.

 

 

Roma Novikov December 25, 2019 at 11:44 AM

can you recheck this with the latest DEV version for pmm1? It's not released yet but we have there already related fix

Thanks

Done

Details

Assignee

Reporter

Priority

Components

Labels

Fix versions

Affects versions

Environment

uname -a
Linux ip-10-9-2-9.ec2.internal 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

 

MySQL RDS v5.6.40

Smart Checklist

Created February 27, 2019 at 10:20 PM
Updated October 9, 2024 at 8:37 AM
Resolved December 27, 2019 at 10:00 AM