Server not showing Prometheus metrics for active agents

General

Escalation

General

Escalation

Description

I have 76 nodes with agents reporting to PMM 2.1.0. There are agent ids for:

PMM Agent
Node exporter
MySQL exporter
Qan MySQL Slowlog Agent

The clients seem to be working and do not report any errors. There do not seem to be any obvious server side errors either, however when I go to Grafana, it only shows 1 system being monitored - the PMM Server itself. The agents are all listed in the PMM Inventory page.

When I first attached clients, I saw 5-6 actually register and show metrics in pmm2, but after restarting them, eventually they all disappeared.

Though I'm not able to see metrics in Prometheus, I am getting QAN slow log analytics from all the nodes, so at least some data is making it through.

How to test

None

How to document

None

Attachments

23 Dec 2019, 09:57 PM

Activity

Show:

Derek Perkins
December 24, 2019 at 4:03 PM

Thank you very much, that helped me to fix my problem. It would be very helpful to show that status in the PMM Inventory page. The agents themselves reported healthy and there was nothing else surfacing that.

Alexey Palazhchenko
December 24, 2019 at 5:36 AM

Please navigate to http(s)://<PMM Server>/prometheus/targets You will see a lot of targets marked as down with error messages like "Get http://uscentral1-keywords-a0-b0-replica-preemptible-0.vitess:42001/metrics?[…] : dial tcp: lookup uscentral1-keywords-a0-b0-replica-preemptible-0.vitess on 10.113.0.10:53: no such host"

Resize issue view side panel

Cannot Reproduce

Details

Assignee

Unassigned

Reporter

Derek Perkins

Priority

Medium

Components

PMM Agent

PMM Server

PMM UI

Affects versions

2.01.0

Environment

Running on Kubernetes: GKE 1.14.9

OS: Google Cloud COS

Bash script that starts the pmm2 agent as each MySQL instance comes up

set -ex

# Redirect pmm-client data to persistent volume
if [ ! -d /vtdataroot/pmm2 ]; then
  FIRST_RUN=1
  mkdir -p /vtdataroot/pmm2
fi

pmm-agent setup --config-file=/vtdataroot/pmm2/pmm-agent.yaml --server-address=pmm2.vitess.svc.cluster.local --server-username=admin --server-password=admin --server-insecure-tls --force "$(hostname).vitess" container "$(hostname)"


# wait for mysql to be available before starting the agent
until [ -e /vtdataroot/tabletdata/mysql.sock ]; do
  echo "Waiting for mysql.sock file"
  sleep 1
done

# start the agent in the background
pmm-agent --config-file=/vtdataroot/pmm2/pmm-agent.yaml &

# give the agent time to start
sleep 5

pmm-admin add mysql --query-source=slowlog --username=nz_monitoring "$(hostname)" 127.0.0.1:3306

# keep the container alive but still responsive to stop requests
trap : TERM INT; sleep infinity & wait

Created December 23, 2019 at 10:10 PM

Updated March 6, 2024 at 5:03 AM

Resolved February 4, 2020 at 8:24 AM