Custom Queries Not Displaying Metrics in GUI for Some RDS Servers
Description
How to test
How to document
Activity
chandra July 26, 2024 at 9:13 AM
Hi
Thanks for your patience,
Please find the below logs for more information.
[root@pmm2-server opt]# time curl -u 'xxxx:xxxxxxxxx' -v http://127.0.0.1:42002/metrics >> mysql_metrics.txt
About to connect() to 127.0.0.1 port 42002 (#0)
Trying 127.0.0.1...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 127.0.0.1 (127.0.0.1) port 42002 (#0)
Server auth using Basic with user 'pmm'
> GET /metrics HTTP/1.1
> Authorization: Basic cG1tOi9hZ2VudF9pZC9iODkyOWM0NS1lMDgxLTRiNmEtOWVmYy0xZGI0OTY4NDQ2ZmE=
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:42002
> Accept: */*
>
0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0< HTTP/1.1 200 OK
< Content-Type: text/plain; version=0.0.4; charset=utf-8
< Date: Fri, 26 Jul 2024 09:11:10 GMT
< Transfer-Encoding: chunked
<
{ [data not shown]
100 39.1M 0 39.1M 0 0 12.6M 0 --:--:-- 0:00:03 --:--:-- 12.6M
Connection #0 to host 127.0.0.1 left intact
real 0m3.120s
user 0m0.064s
sys 0m0.123s
[root@pmm2-server opt]#
Nurlan Moldomurov July 17, 2024 at 5:10 PM
Could you share custom query you added and could you run curl with time in front? It should look like time curl ...
chandra July 17, 2024 at 3:33 PM
Hi
We put custom metrics under the low resolution metrics. We haven’t put under high resolution.
Nurlan Moldomurov July 17, 2024 at 8:49 AM
Hi , Did you put it under high resolution? There is possibility that we can’t collect metric within timeout provided. Could you try to put it into lower resolution?
chandra July 10, 2024 at 11:51 AM
Hi @Roma Novikov,
Thank you for addressing our request. I have already verified the metrics using the "Insight/Advanced Data Exploration" dashboard, but the metrics are not showing.
we are unable to attach a screenshot in the reply.
We have saved a screenshot in my drive and are sharing the link here:
We have added custom queries to our Percona monitoring setup. When executing the API command to fetch metrics for RDS servers, the metrics are available. However, the GUI does not display these metrics in certain instances. This issue is intermittent and occurs only in some places, not universally.
Steps to Reproduce:
Add custom queries to the Percona monitoring setup.
Execute the API command to fetch metrics:
Command: curl -u 'pmm:/agent_id/6ff8410d-258d-4a58-9fc4-cf501efbc9b7' -v http://127.0.0.1:42001/metrics | grep -i -C 10 'mysql_query_p_95_99_latency_min_ms_99'
Checked the GUI for the respective metrics.
Expected Behavior:
The metrics fetched via the API should be consistently displayed in the GUI.
Actual Behavior:
The metrics are not displayed in the GUI for some RDS servers, even though the API command confirms their availability.
Logs:
[root@pmm2-server opt]# curl -u 'pmm:/agent_id/6ff8410d-258d-4a58-9fc4-cf501efbc9b7' -v http://127.0.0.1:42001/metrics | grep -i -C 10 'mysql_query_p_95_99_latency_min_ms_99'
About to connect() to 127.0.0.1 port 42001 (#0)
Trying 127.0.0.1...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 127.0.0.1 (127.0.0.1) port 42001 (#0)
Server auth using Basic with user 'pmm'
> GET /metrics HTTP/1.1
> Authorization: Basic cG1tOi9hZ2VudF9pZC82ZmY4NDEwZC0yNThkLTRhNTgtOWZjNC1jZjUwMWVmYmM5Yjc=
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:42001
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/plain; version=0.0.4; charset=utf-8
< Date: Wed, 03 Jul 2024 11:05:03 GMT
< Transfer-Encoding: chunked
<
{ [data not shown]
100 874k 0 874k 0 0 1006k 0 --:--:-- --:--:-- --:--:-- 1006k
Connection #0 to host 127.0.0.1 left intact
mysql_plugin{library="",license="GPL",name="mysqlx_cache_cleaner",status="ACTIVE",type="AUDIT"} 1
mysql_plugin{library="",license="GPL",name="ngram",status="ACTIVE",type="FTPARSER"} 1
mysql_plugin{library="",license="GPL",name="sha256_password",status="ACTIVE",type="AUTHENTICATION"} 1
mysql_plugin{library="auth_socket.so",license="GPL",name="auth_socket",status="ACTIVE",type="AUTHENTICATION"} 1
mysql_plugin{library="rds_performance_insights.so",license="PROPRIETARY",name="RDS_EVENTS_THREADS_WAITS_CURRENT",status="ACTIVE",type="INFORMATION SCHEMA"} 1
mysql_plugin{library="rds_performance_insights.so",license="PROPRIETARY",name="RDS_PROCESSLIST",status="ACTIVE",type="INFORMATION SCHEMA"} 1
# HELP mysql_query_p_95_99_latency_min_ms_95 P 95 min
# TYPE mysql_query_p_95_99_latency_min_ms_95 gauge
mysql_query_p_95_99_latency_min_ms_95{min_p_95="P95",min_p_99="P99"} 6.31
# HELP mysql_query_p_95_99_latency_min_ms_99 P 99 Min
# TYPE mysql_query_p_95_99_latency_min_ms_99 gauge
mysql_query_p_95_99_latency_min_ms_99{min_p_95="P95",min_p_99="P99"} 144.544
# HELP mysql_sys_schema_auto_increment_usage_details_auto_increment_usage Auto Increment Usage
# TYPE mysql_sys_schema_auto_increment_usage_details_auto_increment_usage gauge
mysql_sys_schema_auto_increment_usage_details_auto_increment_usage{column_name="id",data_type="int",is_signed="Yes",table_name="sbtest1",table_schema="bench_test"} 0.1368
mysql_sys_schema_auto_increment_usage_details_auto_increment_usage{column_name="id",data_type="int",is_signed="Yes",table_name="your_table",table_schema="bench_test"} 0.0001
# HELP mysql_sys_schema_auto_increment_usage_details_current_value Current Value
# TYPE mysql_sys_schema_auto_increment_usage_details_current_value gauge
mysql_sys_schema_auto_increment_usage_details_current_value{column_name="id",data_type="int",is_signed="Yes",table_name="sbtest1",table_schema="bench_test"} 2.937903e+06
mysql_sys_schema_auto_increment_usage_details_current_value{column_name="id",data_type="int",is_signed="Yes",table_name="your_table",table_schema="bench_test"} 1800
# HELP mysql_sys_schema_auto_increment_usage_details_max_value Max Value
# TYPE mysql_sys_schema_auto_increment_usage_details_max_value gauge