Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Force an alert when detecting too many connections

Description

Scenario

The mysqld_exporter cannot connect to the database due to "Too many connections"

Issue

Whilst you can see the issue in the agent logs, you can end up with no data points and this then masks a real issue unless very careful with all alerting rules. The other issue is that with sufficient iterations you can end up with a spattering of data points which also hides the issue.

Improvement

Similar to the "up" metric, perhaps we can have a "inaccessible" metric (mysql_inaccessible etc) that receives a data point with a value of 0 when OK and 1 when down. Alternatively, perhaps the existing mysql_up metric could have a zero datapoint generated instead of a new metric.

This would allow for a single alert rule to check instead of needing to have more complex rules that use absent{} etc

How to test

None

How to document

None

Attachments

1
  • 05 Jun 2020, 11:01 AM

Smart Checklist

Activity

Show:

Lalit Choudhary July 8, 2020 at 9:47 AM

Hi Ceri,

Thank you for the report.

C W June 5, 2020 at 12:05 PM
Edited

To recap, this is a request for a specific metric that is triggered when the exporter is seeing "too many connections" ; OK = 0 , NOK = 1
Scrape errors are unreliable because you may expect some to exist (e.g. TokuDB) or they produce no data (same issue) and so you can't tell if that is no connection at all or not.

C W June 5, 2020 at 11:04 AM

Simply connect to MySQL and issue:

set @v := 0; select count(1) from information_schema.processlist into @v; set global max_connections=@v; select now();

Watch the agent logs with something like:

journalctl -u pmm-agent --since="5 minutes ago" -f | grep -F "Too many"

Review gaps in metrics

Details

Assignee

Reporter

Priority

Components

Labels

Needs QA

Yes

Needs Doc

Yes

Affects versions

Smart Checklist

Created June 5, 2020 at 11:00 AM
Updated November 13, 2024 at 9:50 AM
Configure

Flag notifications