Get pprof data from server component
Description
How to test
How to document
FB test
relates to
Activity
Testing:
Use build perconalab/pmm-server-fb:PR-2560-fbe9c6d
Use Makefile for start pmm-server:
IMAGE := perconalab/pmm-server-fb:PR-2560-fbe9c6d
pull:
docker pull ${IMAGE}
persistence-storage:
docker create --volume /srv \
--name pmm-data \
${IMAGE} /bin/true
up:
docker run --detach \
--publish 443:443 \
--publish 80:80 \
--volumes-from pmm-data \
--name pmm-server \
${IMAGE}
Run pmm-server:
make pull
make persistence-storage
make up
Test:
docker exec pmm-server pmm-admin summary --pprof
Output:
Getting http://127.0.0.1:7777/debug/pprof/profile?seconds=60 ...
Getting http://127.0.0.1:9933/debug/pprof/profile?seconds=60 ...
Getting http://127.0.0.1:7773/debug/pprof/profile?seconds=60 ...
Getting http://127.0.0.1:7777/debug/pprof/heap?gc=1 ...
Getting http://127.0.0.1:7773/debug/pprof/heap?gc=1 ...
Getting http://127.0.0.1:9933/debug/pprof/heap?gc=1 ...
Getting http://127.0.0.1:7773/debug/pprof/trace?seconds=10 ...
Getting http://127.0.0.1:9933/debug/pprof/trace?seconds=10 ...
Getting http://127.0.0.1:7777/debug/pprof/trace?seconds=10 ...
[GET /logs.zip][401] Logs default &{Code:16 Error:Unauthorized Message:Unauthorized}
summary_9b82e17f55a6_2022_06_07_21_54_17.zip created.
docker cp pmm-server:/opt/summary_9b82e17f55a6_2022_06_07_21_54_17.zip .
vim ./summary_9b82e17f55a6_2022_06_07_21_54_17.zip
client/status.json
client/pmm-agent-version.txt
client/pmm-admin-version.txt
client/pmm-agent-config.yaml
client/list.txt
server/pprof/qan-api2/profile.pb.gz
client/pprof/pmm-agent/profile.pb.gz
server/pprof/pmm-managed/profile.pb.gz
client/pprof/pmm-agent/heap.pb.gz
server/pprof/qan-api2/heap.pb.gz
server/pprof/pmm-managed/heap.pb.gz
client/pprof/pmm-agent/trace.out
server/pprof/qan-api2/trace.out
server/pprof/pmm-managed/trace.out
tree ./summary_9b82e17f55a6_2022_06_07_21_54_17/server/
./summary_9b82e17f55a6_2022_06_07_21_54_17/server/
└── pprof
├── pmm-managed
│ ├── heap.pb.gz
│ ├── profile.pb.gz
│ └── trace.out
└── qan-api2
├── heap.pb.gz
├── profile.pb.gz
└── trace.out
3 directories, 6 files
go tool pprof -top ./summary_9b82e17f55a6_2022_06_07_21_54_17/server/pprof/pmm-managed/profile.pb.gz
go tool pprof -top ./summary_9b82e17f55a6_2022_06_07_21_54_17/server/pprof/pmm-managed/heap.pb.gz
go tool pprof -top ./summary_9b82e17f55a6_2022_06_07_21_54_17/server/pprof/qan-api2/profile.pb.gz
go tool pprof -top ./summary_9b82e17f55a6_2022_06_07_21_54_17/server/pprof/qan-api2/heap.pb.gz
go tool trace ./summary_9b82e17f55a6_2022_06_07_21_54_17/server/pprof/pmm-managed/trace.out
go tool trace ./summary_9b82e17f55a6_2022_06_07_21_54_17/server/pprof/qan-api2/trace.out
changing the log level from debug to warn will show a warning message almost always except in the cases when we run it on the pmm-server container. So I suggest limiting this feature to work only on the pmm-server and checking if it's running on the pmm-server here https://github.com/percona/pmm-admin/pull/219/files#diff-a02a9dc132ebd71756372a1aac5b503d0a2b7064a52ce5d6f22180790074e6a8R266.
Since this ticket has more than 2 years I verified pmm-admin code and did some tests.
The source code contains gathering pprof data from server components (pmm-managed, qan-api2) only if pprof endpoints are exposed and reachable by pmm-admin.
If somehow endpoint is not avaliable, the error message is swallowed (not visible for end-user), it just lands in logs.debug. (https://github.com/percona/pmm-admin/blob/main/commands/summary.go#L286)
By the way, pmm-managed has /debug/pprof exposed in devcontainer on localhost interface (127.0.0.1:7773) and even by exposing docker port 7773:7773 i wasn't able to connect to the endpoint.
Changeing interface to 0.0.0.0:7773 solved problem (made changes here: https://github.com/percona/pmm-managed/blob/main/main.go#L104) and finally got pprof data in server directory in summary zip file.
It's not clear for me what else can be done in this task.
May I ask for advice?
I have prepared small fix to make end-user aware of connectivity problems. Shall i create feature build for it also?
https://github.com/percona/pmm-admin/pull/219
Hello @Alexey Palazhchenko,
I'm jira-bot, Percona's automated helper script. Your bug report is important
to us but we've been unable to reproduce it, and asked you for more
information. If we haven't heard from you on this in 3 more weeks, the issue
will be automatically closed.
Use case:
pmm-admin summary --pprof should contain pprof data from server
Acceptance criteria:
summary should contain pprof data from server
pmm-admin should be run on pmm-server
UI/UX:
$ pmm-admin summary --pprof
How to test:
1. Login on PMM-SERVER
2. Run
$ pmm-admin summary --pprof
3. Verify result file
Out of scope:
Gathering server's pprof data outside pmm-server will be delivered in https://perconadev.atlassian.net/browse/PMM-5492#icft=PMM-5492