pmm-agent log-level match exporters output
Description
How to test
How to document
FB test
Activity
Testing build
mkdir -p ~/go/src/github.com/YaroslavPodorvanov/pmm/example/PMM-10122
cd ~/go/src/github.com/YaroslavPodorvanov/pmm/example/PMM-10122
wget https://s3.us-east-2.amazonaws.com/pmm-build-cache/PR-BUILDS/pmm2-client/pmm2-client-PR-2639-e82d082.tar.gz
tar -xzvf pmm2-client-PR-2639-e82d082.tar.gz pmm2-client-2.30.0/bin/
tree ./pmm2-client-2.30.0/
./pmm2-client-2.30.0/
└── bin
├── azure_exporter
├── clickhouse_exporter
├── mongodb_exporter
├── mysqld_exporter
├── node_exporter
├── pmm-admin
├── pmm-agent
├── pmm-agent-entrypoint
├── pmm-managed
├── pmm-managed-init
├── pmm-managed-starlark
├── postgres_exporter
├── proxysql_exporter
├── pt-mongodb-summary
├── pt-mysql-summary
├── pt-pg-summary
├── pt-summary
├── rds_exporter
└── vmagent
1 directory, 19 files
pmm-agent --version
./pmm2-client-2.30.0/bin/pmm-agent --version
ProjectName: pmm-agent
Version: 2.30.0-HEAD-e82d0825
PMMVersion: 2.30.0-HEAD-e82d0825
Timestamp: 2022-07-14 20:41:34 (UTC)
FullCommit: 0300e507d5b2d1583b8e94a54bbcca8dbe24690f
./pmm2-client-2.30.0/bin/pmm-admin --version
ProjectName: pmm-admin
Version: 2.30.0-HEAD-e82d0825
PMMVersion: 2.30.0-HEAD-e82d0825
Timestamp: 2022-07-14 20:41:34 (UTC)
FullCommit: 0300e507d5b2d1583b8e94a54bbcca8dbe24690f
pmm-server
Makefile:
IMAGE := perconalab/pmm-server-fb:PR-2639-e82d082
pull:
docker pull ${IMAGE}
persistence-storage:
docker create --volume /srv \
--name pmm-data \
${IMAGE} /bin/true
version:
curl -ku admin:admin https://localhost/v1/version
docker exec pmm-server pmm-managed --version
up:
docker run --detach \
--publish 443:443 \
--publish 80:80 \
--volumes-from pmm-data \
--name pmm-server \
${IMAGE}
make pull persistence-storage up
make version
curl -ku admin:admin https://localhost/v1/version
{
"version": "2.30.0",
"server": {
"version": "2.30.0",
"full_version": "1:2.30.0-19.2207142015.0300e50.el7",
"timestamp": "2022-07-14T20:16:38Z"
},
"managed": {
"version": "2.30.0-HEAD-bd4a13c7",
"full_version": "0300e507d5b2d1583b8e94a54bbcca8dbe24690f",
"timestamp": "2022-07-14T20:15:34Z"
},
"distribution_method": "DOCKER"
}
docker exec pmm-server pmm-managed --version
ProjectName: pmm-managed
Version: 2.30.0-HEAD-bd4a13c7
PMMVersion: 2.30.0-HEAD-bd4a13c7
Timestamp: 2022-07-14 20:15:34 (UTC)
FullCommit: 0300e507d5b2d1583b8e94a54bbcca8dbe24690f
pmm-agent setup and run [Terminal 1]
./pmm2-client-2.30.0/bin/pmm-agent setup \
--config-file=pmm-agent-conf.yml \
--server-insecure-tls \
--server-address=127.0.0.1 \
--server-username=admin \
--server-password=admin \
--paths-base=./pmm2-client-2.30.0/bin/ \
--paths-exporters_base=./pmm2-client-2.30.0/bin/
./pmm2-client-2.30.0/bin/pmm-agent --config-file=pmm-agent-conf.yml --log-level=info 2>&1 | tee logs.txt
MySQL exporter logs:
./pmm2-client-2.30.0/bin/pmm-admin add mysql --username=pmm-agent --password=pmm-agent-password --query-source=perfschema --log-level=info
pmm-agent output success match:
INFO[2022-07-15T03:20:00.001+03:00] Sending status: WAITING. agentID=/agent_id/cf08fd8e-34ff-4d63-8662-4981f970ed77 component=agent-builtin type=qan_mysql_perfschema_agent
ERRO[2022-07-15T03:20:02.273+03:00] failed to query events_statements_history: Error 1142: SELECT command denied to user 'pmm-agent'@'172.27.0.1' for table 'events_statements_history' agentID=/agent_id/cf08fd8e-34ff-4d63-8662-4981f970ed77 component=agent-builtin type=qan_mysql_perfschema_agent
PostgreSQL exporter logs:
./pmm2-client-2.30.0/bin/pmm-admin add postgresql --username=pmm-agent --password=pmm-agent-password --query-source=pgstatements --log-level=info
pmm-agent output success match:
INFO[2022-07-15T03:26:13.593+03:00] time="2022-07-15T03:26:13+03:00" level=info msg="Semantic Version Changed on \"127.0.0.1:5432\": 0.0.0 -> 11.15.0" source="postgres_exporter.go:1520" agentID=/agent_id/f8955fc1-4bcc-41ee-979a-8e55da356619 component=agent-process type=postgres_exporter
ERRO[2022-07-15T03:26:13.593+03:00] time="2022-07-15T03:26:13+03:00" level=error msg="failed read dir \"/home/senseye/go/src/github.com/YaroslavPodorvanov/pmm/example/PMM-10122/pmm2-client-2.30.0/bin/collectors/custom-queries/postgresql/medium-resolution\" for custom query. reason: open /home/senseye/go/src/github.com/YaroslavPodorvanov/pmm/example/PMM-10122/pmm2-client-2.30.0/bin/collectors/custom-queries/postgresql/medium-resolution: no such file or directory" source="postgres_exporter.go:1570" agentID=/agent_id/f8955fc1-4bcc-41ee-979a-8e55da356619 component=agent-process type=postgres_exporter
MongoDB exporter logs:
./pmm2-client-2.30.0/bin/pmm-admin add mongodb --username=root --password=root-password --query-source=profiler --log-level=info
pmm-agent output success match:
INFO[2022-07-15T03:27:59.468+03:00] Sending status: RUNNING (port 42005). agentID=/agent_id/da86eddc-a735-4b6a-b4b4-b0a7ef87aa89 component=agent-process type=mongodb_exporter
WARN[2022-07-15T03:28:17.604+03:00] time="2022-07-15T03:28:17+03:00" level=warning msg="cannot create metrics for oplog: mongo: no documents in result" agentID=/agent_id/da86eddc-a735-4b6a-b4b4-b0a7ef87aa89 component=agent-process type=mongodb_exporter
ProxySQL exporter logs:
./pmm2-client-2.30.0/bin/pmm-admin add proxysql --log-level=info --skip-connection-check
pmm-agent output success match:
INFO[2022-07-15T03:30:55.479+03:00] Sending status: RUNNING (port 42006). agentID=/agent_id/e9a38627-7882-439d-a473-f500cfb2144d component=agent-process type=proxysql_exporter
ERRO[2022-07-15T03:32:07.980+03:00] time="2022-07-15T03:32:07+03:00" level=error msg="Error opening connection to ProxySQL: dial tcp 127.0.0.1:6032: connect: connection refused" source="exporter.go:169" agentID=/agent_id/e9a38627-7882-439d-a473-f500cfb2144d component=agent-process type=proxysql_exporter
@Beata Handzelova, in all test cases i found step "Check pmm-agent logs" that cover this issue. Great.
@Yaroslav Podorvanov thanks! I updated the existing linked test cases instead of checklist. Can you please check if they reflect reality (and edit if needed)?
@Beata Handzelova, so this is pmm-agent output:
INFO[2022-05-31T17:23:37.620+03:00] time="2022-05-31T17:23:37+03:00" level=error msg="cannot scrape status with a chosen query \"SHOW SLAVE STATUS\": Error 1227: Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation" source="slave_status.go:104" agentID=/agent_id/6d89c104-ff8e-4af8-ac11-e5c948b831a4 component=agent-process type=mysqld_exporter
Main problem that error from mysqld_exporter log as INFO
After this issue error from mysqld_exporter will be log as ERROR
Sorry, I don't even understand what was done in this task. It's just links to different tasks and everything is in "out of scope". Please provide better description of what was done here and what's expected result. TIA.
User story:
As a PMM user, i want to apply pmm-agent --log-level to exporters output
So each message from exporter with log-level=WARN will logged in pmm-agent with same log-level=WARN
UI/UX:
> pmm-agent --log-level=WARN
Out of scope (explain):
In https://jira.percona.com/browse/PMM-9840 added --log-level to exporters
In https://jira.percona.com/browse/PMM-5668 added --log-level to pmm-agent
In https://jira.percona.com/browse/PMM-7326 added --log-level to pmm-admin to configure exporters --log-level like
> pmm-admin add mysql --log-level=info
Out of scope (now):
Now all messages from exporter logged in pmm-agent with log-level=INFO
When pmm-agent --log-level=WARN then exporter output gone
Original report from @Artem Gavrilov after DEMO:
INFO[2022-05-31T17:23:37.620+03:00] time="2022-05-31T17:23:37+03:00" level=error msg="cannot scrape status with a chosen query \"SHOW SLAVE STATUS\": Error 1227: Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation" source="slave_status.go:104" agentID=/agent_id/6d89c104-ff8e-4af8-ac11-e5c948b831a4 component=agent-process type=mysqld_exporter