Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

RDS_exporter crashes when DB is stopped

Description

Steps to reproduce:

1. Stop RDS instance listed in configuration file config.yml
2. Launch RDS_exporter

Actual result:  when RDS instance is stopped/not available

panic error.

pmm2/exporters# ./rds_exporter pmm2/exporters# ./rds_exporter INFO[0000] Starting RDS exporter (version=, branch=, revision=)  source="main.go:30"INFO[0000] Build context (go=go1.12.12, user=, date=)    source="main.go:31"INFO[0000] Creating sessions...                          component=sessions source="transport.go:51"ERRO[0000] Failed to get resource IDs: RequestError: send request failedcaused by: Post https://rds.us-east-1a.amazonaws.com/: dial tcp: lookup rds.us-east-1a.amazonaws.com on 127.0.0.53:53: no such host.  component=sessions source="sessions.go:102"ERRO[0000] Skipping instance us-east-1a/lalit-pmm2-bug - can't determine resourceID.  component=sessions source="sessions.go:125"Region  Instance  Resource ID  IntervalINFO[0000] Using 0 sessions.                             component=sessions source="sessions.go:149"panic: collector has no descriptors goroutine 1 [running]:github.com/percona/rds_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc00002ec80, 0xc00018df18, 0x1, 0x1) /tmp/go/src/github.com/percona/rds_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:404 +0xadmain.main() /tmp/go/src/github.com/percona/rds_exporter/main.go:58 +0x894

 

Expected result:

  • It should give a valid error that RDS instance is not reachable / stopped  instead of panic error

  • Continue to monitoring other RDS  instances

 

How to test

None

How to document

None

Activity

Show:

Lalit Choudhary 
December 25, 2019 at 1:46 PM

With PMM 2.2.0, I do not see the panic error anymore when rds instance stopped/not reachable.

# ./rds_exporter INFO[0000] Starting RDS exporter (version=, branch=, revision=) source="main.go:30" INFO[0000] Build context (go=go1.12.12, user=, date=) source="main.go:31" INFO[0000] Creating sessions... component=sessions source="sessions.go:45" ERRO[0000] Skipping us-east-1f/pmm2test - can't determine resourceID. component=sessions source="sessions.go:137" Region Instance Resource ID Interval INFO[0000] Using 0 sessions. component=sessions source="sessions.go:161" INFO[0000] Basic metrics : http://:9042/basic source="main.go:65" INFO[0000] Enhanced metrics: http://:9042/enhanced source="main.go:66"

 

 

Roma Novikov 
December 25, 2019 at 11:46 AM

pls recheck this against recently relesed pmm 2.2.0. We have related fix there.
Thanks

Lalit Choudhary 
December 23, 2019 at 12:03 PM

Hi Viktor,

Thank you for the report. Validated is as described.

 

Steps to reproduce:

1. Stop RDS instance listed in configuration file config.yml
2. Launch RDS_exporter

Actual result:  when RDS instance is stopped/not available

panic error.

pmm2/exporters# ./rds_exporter pmm2/exporters# ./rds_exporter INFO[0000] Starting RDS exporter (version=, branch=, revision=)  source="main.go:30"INFO[0000] Build context (go=go1.12.12, user=, date=)    source="main.go:31"INFO[0000] Creating sessions...                          component=sessions source="transport.go:51"ERRO[0000] Failed to get resource IDs: RequestError: send request failedcaused by: Post https://rds.us-east-1a.amazonaws.com/: dial tcp: lookup rds.us-east-1a.amazonaws.com on 127.0.0.53:53: no such host.  component=sessions source="sessions.go:102"ERRO[0000] Skipping instance us-east-1a/lalit-pmm2-bug - can't determine resourceID.  component=sessions source="sessions.go:125"Region  Instance  Resource ID  IntervalINFO[0000] Using 0 sessions.                             component=sessions source="sessions.go:149"panic: collector has no descriptors goroutine 1 [running]:github.com/percona/rds_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc00002ec80, 0xc00018df18, 0x1, 0x1) /tmp/go/src/github.com/percona/rds_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:404 +0xadmain.main() /tmp/go/src/github.com/percona/rds_exporter/main.go:58 +0x894

 

Expected result:

It should give a valid error that RDS instance is not reachable / stopped  instead of panic error

 

Done

Details

Assignee

Reporter

Priority

Components

Labels

Fix versions

Affects versions

Created November 20, 2019 at 8:13 AM
Updated November 13, 2024 at 9:47 AM
Resolved December 25, 2019 at 1:47 PM

Flag notifications