If 1 RDS instance doesn't have logs or enhanced monitoring turned on all RDS collection fails

Description

I recently started working on these existing MySQL RDS instances, which don't have Enhanced monitoring turned on, some have no logs going to file or Cloud Watch. (yeah I know)

Unfortunately adding any of these instances to PMM causes ALL the RDS CloudWatch metrics collection to fail. 

  • All the graphs at the bottom of the MySQL Overview show nothing

  • Cloud Amazon RDS / Aurora MySQL Metrics Grafana page displays a red message with  Templating init failed - Metric request error

  • The pmm-rds_exporter-9042.service crashes and restarts endlessly

I have confirmed this by using a test instance of PMM and adding to it just 1 host which I know has Enhanced monitoring on & logs enabled.  With that setup I get graphs at the bottom - but I still get the Templating init failed.

This should be more robust and collect the metrics that it can. CloudWatch (non-enhanced) is showing metrics in the AWS console, so there is a lot which can be pulled down anyway.

 

  1.  

    # /var/log/pmm-rds_exporter-9042.log time="2019-02-27T20:08:10Z" level=info msg="Starting RDS exporter (version=, branch=, revision=)" source="main.go:30" time="2019-02-27T20:08:10Z" level=info msg="Build context (go=go1.10.1, user=, date=)" source="main.go:31" time="2019-02-27T20:08:10Z" level=info msg="Creating sessions..." component=sessions source="sessions.go:35" Region Instance Resource ID Interval us-east-1 db-prod db-Axxxxxxxxxxxxxxxxxxxxxxxxx 0s us-east-1 db-prod-read-east-1d db-Fxxxxxxxxxxxxxxxxxxxxxxxxxx 0s us-east-1 db-staging db-Hxxxxxxxxxxxxxxxxxxxxxxxxxx 1m0s us-west-1 db-prod-read-west-1 db-Cxxxxxxxxxxxxxxxxxxxxxxxxxx 0s time="2019-02-27T20:08:11Z" level=info msg="Using 2 sessions." component=sessions source="sessions.go:149" time="2019-02-27T20:08:11Z" level=info msg="Updating enhanced metrics every 10s." component=enhanced source="collector.go:49" time="2019-02-27T20:08:11Z" level=error msg="Failed to filter log events: ResourceNotFoundException: The specified log stream does not exist.\n\tstatus code: 400, request id: 684debf0-3acb-11e9-a2f8-ada7e9bd4a87." component=enhanced source="scraper.go:109" time="2019-02-27T20:08:11Z" level=info msg="Updating enhanced metrics every 10s." component=enhanced source="collector.go:49" time="2019-02-27T20:08:11Z" level=error msg="Failed to filter log events: ResourceNotFoundException: The specified log group does not exist.\n\tstatus code: 400, request id: 68723d54-3acb-11e9-898d-b1893eccff58." component=enhanced source="scraper.go:109" panic: collector has no descriptorsgoroutine 1 [running]: github.com/percona/rds_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc420589c00, 0xc4201c3ea8, 0x1, 0x1) /home/builder/rpm/BUILD/rds_exporter-e71e0f3b4f101cf84191052834e3af3d774a0aa9/src/github.com/percona/rds_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:404 +0x9e main.main() /home/builder/rpm/BUILD/rds_exporter-e71e0f3b4f101cf84191052834e3af3d774a0aa9/src/github.com/percona/rds_exporter/main.go:58 +0x570
  1.  

    # /var/log/messages Feb 27 22:09:11 ip-10-9-2-9 systemd: pmm-rds_exporter-9042.service holdoff time over, scheduling restart. Feb 27 22:09:11 ip-10-9-2-9 systemd: Started pmm-rds_exporter-9042. Feb 27 22:09:11 ip-10-9-2-9 systemd: Starting pmm-rds_exporter-9042... Feb 27 22:09:11 ip-10-9-2-9 systemd: pmm-rds_exporter-9042.service: main process exited, code=exited, status=2/INVALIDARGUMENT Feb 27 22:09:11 ip-10-9-2-9 systemd: Unit pmm-rds_exporter-9042.service entered failed state. Feb 27 22:09:11 ip-10-9-2-9 systemd: pmm-rds_exporter-9042.service failed.

 

 

How to test

None

How to document

None

AFFECTED CS IDs

273107

Attachments

1
  • 27 Dec 2019, 09:52 AM

Smart Checklist

Activity

Lalit Choudhary February 16, 2021 at 1:02 PM

Lalit Choudhary December 22, 2020 at 7:05 AM

Please open the new report for the 2.12 version issue with bug details.

you can mention this bug in reference. 

December 21, 2020 at 3:54 PM

this is still occurring on PMM 2.12

Lalit Choudhary December 27, 2019 at 9:56 AM
Edited

Tested with 1.17.3-Dev ,

added 2 rds instances 1 with enhanced monitoring enabled and another one with enhanced monitoring Disable.

Restarted pmm-server

time="2019-12-27T09:13:40Z" level=info msg="Starting RDS exporter (version=, branch=, revision=)" source="main.go:30" time="2019-12-27T09:13:40Z" level=info msg="Build context (go=go1.10.1, user=, date=)" source="main.go:31" time="2019-12-27T09:13:40Z" level=info msg="Creating sessions..." component=sessions source="sessions.go:45" Region Instance Resource ID Interval us-east-1 pmm-rds-bugtest db-SC6GC23Y5X5HPMQNI4TU6ZZ34U 1m0s us-east-1 pmm-rds-bugtest-2 db-KNIGMVH3HGNA4JBFV2FX2LN4AY 0s time="2019-12-27T09:13:41Z" level=info msg="Using 1 sessions." component=sessions source="sessions.go:161" time="2019-12-27T09:13:41Z" level=info msg="Updating enhanced metrics every 10s." component=enhanced source="collector.go:49" time="2019-12-27T09:13:41Z" level=error msg="Failed to filter log events: ResourceNotFoundException: The specified log stream does not exist.\n\tstatus code: 400, request id: e62e4638-347a-4bc1-a8f4-9eb913ad8db7." component=enhanced source="scraper.go:109" time="2019-12-27T09:13:41Z" level=info msg="Basic metrics : http://127.0.0.1:9042/basic" source="main.go:65" time="2019-12-27T09:13:41Z" level=info msg="Enhanced metrics: http://127.0.0.1:9042/enhanced" source="main.go:66" time="2019-12-27T09:13:51Z" level=error msg="Failed to filter log events: ResourceNotFoundException: The specified log stream does not exist.\n\tstatus code: 400, request id: 86012848-16c6-442d-861a-b26ffda09f9a." component=enhanced source="scraper.go:109" time="2019-12-27T09:14:01Z" level=error msg="Failed to filter log events: ResourceNotFoundException: The specified log stream does not exist.\n\tstatus code: 400, request id: 34a682a6-bb94-43c4-a4b5-5c819102111d." component=enhanced source="scraper.go:109"
  1. I do not see a panic error in rds_exporter logs

  2. rds_exporter running fine

  3. it has error since one of the rds instance not enabled with monitoring. 

  4. And for rds instance with monitoring enabled we can see all graphs 

The only problem here for keeping rds instance which has enhanced monitoring Disabled is, it will keep adding error message for  "component=enhanced source="scraper.go" every 10 seconds. 

Removing such RDS instance from monitoring will fix this issue.

 

 

Roma Novikov December 25, 2019 at 11:44 AM

can you recheck this with the latest DEV version for pmm1? It's not released yet but we have there already related fix

Thanks

Done

Details

Assignee

Reporter

Priority

Components

Labels

Fix versions

Affects versions

Environment

uname -a
Linux ip-10-9-2-9.ec2.internal 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

 

MySQL RDS v5.6.40

Smart Checklist

Created February 27, 2019 at 10:20 PM
Updated October 9, 2024 at 8:37 AM
Resolved December 27, 2019 at 10:00 AM