scrape_duration_seconds is broken with current prometheus configuration file
Description
How to test
How to document
Attachments
Confluence content
Activity
Alexey Palazhchenko July 15, 2019 at 11:03 AM
Done as a part of other PMM 2.0 tasks.
Mykola Marzhan September 27, 2017 at 10:23 AM
job relabeling can be removed.
instance relabeling is needed to keep human readable hosts in grafana
Roma Novikov July 17, 2017 at 3:31 PM
It seems like now we don't use job name in graphs and the main problem we can face if we remove relabeling: 2 metrics on graphs while we'll see interval with 2 approaches (with relabeling and without). Also possible problems in some label/cluster detection.
So we can test this idea to see all problems and decide when to implement it.
Kenny Gryp July 17, 2017 at 2:38 PM
: My question was aimed at who mentioned that we should keep on using the relabeling to job=mysql as we would presumably need it for all/many graph creation. That's how I understood it.
You seem to mention graph creation of scrape interval seconds. Yes... that's why i created ... we need to keep the original labels to graph scrape interval seconds.
Alexey Palazhchenko July 17, 2017 at 1:34 PM
Kenny, I was answering your latest question. Graph designer needs to use job name if s/he wants to show scrape duration of mysqld exporter(s), not some other export (instance may be reused in a dynamic environment). If we remove relabeling, then we should change job="mysql" to job=~"mysql.+".
We don't use job names for graph creation as far as I know
Then we should.
I don't think anyone still knows why we used relabeling in the first place there. We could remove it.
You can see
prometheus.ymlrelabels the 3 mysql jobs tomysql:This means that when you try to investigate why some data is missing, you usually go to look at
scrape_duration_seconds. however, the relabeling causes the job name to be mysql for all fetches, making this metric unusable for all 3 mysql scrapesCan we skip the relabeling?
The following graphs will be broken if we remove job relabeling for all metrics. We can fix them, but historical data will be unaccessible:
InnoDB Log Buffer Performance
MySQL Uptime
Current QPS
InnoDB Buffer Pool Size
Buffer Pool Size of Total RAM
MySQL Internal Memory Overview
Query Response Time Distribution
Read Query Response Time Distribution
Write Query Response Time Distribution
Read Only
TokuDB Cache Size