Exporter CPU Usage glitches

Description

Hi,

https://pmmdemo.percona.com/graph/dashboard/db/prometheus-exporters-overview?orgId=1

These spikes to thosands percent do not look right


Peter Zaitsev, CEO, Percona

Percona Live 2018 Open Source Database Conference
<https://www.percona.com/live/18/> April 23 - 25, 2018 Santa Clara,CA Get
Your Ticket Now!, Inquire about Sponsorship!

How to test

None

How to document

None

Attachments

6

Activity

Roma Novikov 
March 25, 2018 at 7:50 PM

, yes , merged 

Nailya Kutlubaeva 
March 24, 2018 at 7:18 PM

is is ready for testing?

Kamil Dziedzic 
March 23, 2018 at 3:35 PM
(edited)

could you confirm that this works correctly? http://18.219.12.208/graph/dashboard/db/prometheus-exporters-overview?refresh=1m&orgId=1

 

You can run your own instance with the following:

docker - perconalab/pmm-server-fb:PR-11-31a88a2
client - https://s3.us-east-2.amazonaws.com/pmm-build-cache/pmm-client/pmm-client-PR-11-31a88a2.tar.gz

https://github.com/Percona-Lab/pmm-submodules/pull/11#issuecomment-375582654

 

p.s. you are the best winking face Thanks for feature builds winking face

 

Vadim Yalovets 
March 15, 2018 at 8:20 AM

The original formula is absolutely correct due to the fact that
metrics process_cpu_seconds_total are taken from /proc/../stat files.

...
// CPUTime returns the total CPU user and system time in seconds.
func (s ProcStat) CPUTime() float64 {
return float64(s.UTime+s.STime) / userHZ
}
...

I have compared amount of process_cpu_seconds_total scrapers in 1.7.x and 1.8.x
So it looks like the reason of the issue is additional, duplicate values from mysqld_exporter.
The new exporter are managed to collect 71 or 72 values in 60 seconds.
IMHO exporter scraps this parameter in each resolution
(hr -> 59) + (mr -> 11) + (lr -> 1)

1.7.x

1.8.x

Peter Zaitsev 
March 14, 2018 at 11:11 PM
(edited)

Folks,

 

Even if MySQLD exporter is using CPU more intensively it can't be 5000% 

These graphs show the CPU usage  by SPECIFIC exporter or all exporters,   node_cpu  shows ALL CPU usage on the system

While in some particular cases of the idle system these numbers may match they are not the same. 

I strongly suggest to identify why proces_cpu  metrics are off and fix the issue instead of looking for some strange workarounds 

Only if it is absolutely impossible we should consider "workarounds"

Done

Details

Assignee

Reporter

Priority

Fix versions

Affects versions

Created March 11, 2018 at 11:14 AM
Updated November 13, 2024 at 9:45 AM
Resolved April 4, 2018 at 2:47 PM

Flag notifications