Exporter CPU Usage glitches
Description
How to test
How to document
Attachments
- 15 Mar 2018, 08:16 AM
- 15 Mar 2018, 08:16 AM
- 14 Mar 2018, 11:35 AM
- 12 Mar 2018, 08:33 PM
- 11 Mar 2018, 12:27 PM
- 11 Mar 2018, 11:14 AM
Confluence content
Activity
Roma Novikov March 25, 2018 at 7:50 PM
@Nailya Kutlubaeva, yes , merged
Nailya Kutlubaeva March 24, 2018 at 7:18 PM
@Kamil Dziedzic is is ready for testing?
Kamil Dziedzic March 23, 2018 at 3:35 PM(edited)
@Vadim Yalovets could you confirm that this works correctly? http://18.219.12.208/graph/dashboard/db/prometheus-exporters-overview?refresh=1m&orgId=1
You can run your own instance with the following:
docker - perconalab/pmm-server-fb:PR-11-31a88a2
client - https://s3.us-east-2.amazonaws.com/pmm-build-cache/pmm-client/pmm-client-PR-11-31a88a2.tar.gz
https://github.com/Percona-Lab/pmm-submodules/pull/11#issuecomment-375582654
p.s. @Mykola Marzhan you are the best Thanks for feature builds
Vadim Yalovets March 15, 2018 at 8:20 AM
The original formula is absolutely correct due to the fact that
metrics process_cpu_seconds_total are taken from /proc/../stat files.
...
// CPUTime returns the total CPU user and system time in seconds.
func (s ProcStat) CPUTime() float64 {
return float64(s.UTime+s.STime) / userHZ
}
...
I have compared amount of process_cpu_seconds_total scrapers in 1.7.x and 1.8.x
So it looks like the reason of the issue is additional, duplicate values from mysqld_exporter.
The new exporter are managed to collect 71 or 72 values in 60 seconds.
IMHO exporter scraps this parameter in each resolution
(hr -> 59) + (mr -> 11) + (lr -> 1)
1.7.x | 1.8.x |
---|---|
Peter Zaitsev March 14, 2018 at 11:11 PM(edited)
Folks,
Even if MySQLD exporter is using CPU more intensively it can't be 5000%
These graphs show the CPU usage by SPECIFIC exporter or all exporters, node_cpu shows ALL CPU usage on the system
While in some particular cases of the idle system these numbers may match they are not the same.
I strongly suggest to identify why proces_cpu metrics are off and fix the issue instead of looking for some strange workarounds
Only if it is absolutely impossible we should consider "workarounds"
Hi,
https://pmmdemo.percona.com/graph/dashboard/db/prometheus-exporters-overview?orgId=1
These spikes to thosands percent do not look right
–
Peter Zaitsev, CEO, Percona
Percona Live 2018 Open Source Database Conference
<https://www.percona.com/live/18/> April 23 - 25, 2018 Santa Clara,CA Get
Your Ticket Now!, Inquire about Sponsorship!