Agents stuck in Starting after upgrading PMM Server with +100 services
General
Escalation
General
Escalation
Description
How to test
None
How to document
None
Attachments
1
- 13 Aug 2021, 01:00 PM
Smart Checklist
Activity
Show:
Nurlan Moldomurov August 26, 2021 at 12:35 PM
@Beata Handzelova, could you please add pmm-managed and pmm-agent logs in DEBUG mode?
Nailya Kutlubaeva August 19, 2021 at 8:04 AM
I tried to reproduce this issue on Ubuntu with 100 services but couldn't, everything works fine
Nailya Kutlubaeva August 13, 2021 at 4:39 PM
we had very similar bug https://jira.percona.com/browse/PMM-7431
Cannot Reproduce
Created August 13, 2021 at 12:59 PM
Updated March 6, 2024 at 2:19 AM
Resolved August 30, 2021 at 10:40 AM
STR:
Start PMM Server and client 2.20.0
Add +100 instances to monitoring
for i in {1..100} ; do pmm-admin add mysql --username=root --password=pass --query-source=perfschema node_$i mysql8:3306 & sleep 3; doneUpgrade PMM Server to RC
Check pmm-admin list or Inventory page
Result:
Agent type Status Metrics Mode Agent ID Service ID pmm_agent Connected /agent_id/df330d90-48fd-4db4-a456-a25c171f52ab node_exporter Starting push /agent_id/c4becb17-7068-4fe6-8358-5fceba05f484 mysqld_exporter Starting push /agent_id/018674c9-0009-475d-9ed4-fcb28fb3f11b /service_id/2c5bfbf3-a68f-4953-84e3-968abbf13c67 mysqld_exporter Starting push /agent_id/08367002-266e-458d-835e-33a527b9c768 /service_id/16f9c61b-0f6c-488a-b5a8-06f94e7e558c mysqld_exporter Starting push /agent_id/09c3c0dc-5cef-42a9-9e6e-7be91f8f977c /service_id/9a3aba6c-a9d6-4c6f-9941-76709e90a241 mysqld_exporter Starting push /agent_id/0ceccfee-3a38-491c-bf62-916a2cb212ed /service_id/b70a3649-a813-428e-b266-8044b8048395 mysqld_exporter Starting push /agent_id/1460abaa-5c8f-448b-b144-6ae354e97dbc /service_id/0188b0a8-d233-4dfb-a0bc-57917fea84d8 mysqld_exporter Starting push /agent_id/15420fb2-5b80-473b-b0d2-6c2cdd294c41 /service_id/21927fa6-161a-429c-8be3-6f5b1d4f29dc mysqld_exporter Starting push /agent_id/15ef83c4-8333-43ed-9f3c-309b454e4112 /service_id/f0150ba2-9a16-4fe0-a086-ae43da7a6804 mysqld_exporter Starting push /agent_id/1846e067-a69d-47c0-9e9b-1ad8f4539145 /service_id/f918fd4c-10a1-4110-b829-f0f789eda5ba mysqld_exporter Starting push /agent_id/19240552-8753-44c9-a346-258c26048d43 /service_id/235665cf-6155-4098-b77b-8b301240df14 mysqld_exporter Starting push /agent_id/1a5f22ab-5462-498a-805d-bf4f5889afb1 /service_id/5feea6cd-5d32-458f-9ce3-350cdc2e8f25 mysqld_exporter Starting push /agent_id/1b4ce295-4919-44a5-9acd-1e27400db50b /service_id/646a0d5b-4f6a-4a00-88be-d4ed0955b3b6 mysqld_exporter Starting push /agent_id/1c58fd47-dfd1-4423-a8da-8715b9327c96 /service_id/1db5dd10-8d96-451d-9a26-6969871f93ad mysqld_exporter Starting push /agent_id/1cc2d8dc-bf37-4e63-be5f-a5503ae104ce /service_id/5107dd70-dbd5-4e8b-b0ec-22370349045e mysqld_exporter Starting push /agent_id/1cc6565c-4531-4c7e-b4f8-cbc1f62c1cd7 /service_id/621cef6e-0283-45f4-8f1a-e6a1c161f441 mysqld_exporter Starting push /agent_id/1cf02f9f-870a-482c-bfe1-c9c130f211c0 /service_id/3e58842a-ce49-4d3b-b187-5e1d407c3d3c mysqld_exporter Starting push /agent_id/2478c3a3-b259-4fc5-986d-25f5fc081ff4 /service_id/2e92728d-3426-4948-b9e4-3ad2d0fbd865 mysqld_exporter Starting push /agent_id/25e4eb1e-6423-42dc-8c79-ec33c14e1354 /service_id/037194d2-2093-47de-9344-82a460992a48 mysqld_exporter Starting push /agent_id/2aa336a7-8234-4adb-a2e9-2d31c6f52bf6 /service_id/df80b5d3-fd1c-42f4-86c8-ac89f98c355d mysqld_exporter Starting push /agent_id/2b9ee86e-8448-479a-b51e-a86e7ef27ec8 /service_id/d8a151a6-ef8c-488a-88d9-abf0b2f88996 mysqld_exporter Starting push /agent_id/2bbdd41f-af18-490c-bac5-1a4b1bc32b4d /service_id/e863e544-1e68-40ef-a945-1bf594b10cee mysqld_exporter Starting push /agent_id/2cdb95d0-03c5-48d9-ad63-60f8b6da7bf3 /service_id/83be5d84-e34e-4e5c-ab21-b4261431eba7 mysqld_exporter Starting push /agent_id/2d21b44b-dc6f-4a51-afc7-a663dc637a36 /service_id/1737d3fd-6bba-48e5-ad96-955ce3569644 mysqld_exporter Starting push /agent_id/30b68130-b933-42d0-af83-6a64d1c13fd5 /service_id/2470d3ef-c583-457f-9996-1bc6953b6956 mysqld_exporter Starting push /agent_id/37594fc6-2b3b-41f9-8ae6-37ac4045b305 /service_id/246a40b0-e539-4e8d-853f-7498cdbc83cc mysqld_exporter Starting push /agent_id/3bec36a7-f11f-41ee-961c-30e048ca62f2 /service_id/00130b9a-e0c9-4b88-b13f-67eef9da6fa9 mysqld_exporter Starting push /agent_id/4662235d-e6df-4df9-8ded-daf8ad89f914 /service_id/2d27d91a-71f0-4751-9a66-e5533cfe6795 mysqld_exporter Starting push /agent_id/4c40e60f-58d4-42da-ac40-6df91169a910 /service_id/c4825858-e7be-4714-928d-5d4ba567bbd2 mysqld_exporter Starting push /agent_id/4f5d192c-f671-495a-ae0f-e5d9e12958e5 /service_id/f4b458e0-b174-4ec6-be55-860ec4b9c0a7 mysqld_exporter Starting push /agent_id/5621ad15-dfbe-46e4-ae72-c97543f973a0 /service_id/3ea495ba-5453-457e-9852-b5beb2340f91 mysqld_exporter Starting push /agent_id/5b0de9e8-4079-44ee-b53b-65a5c7ade019 /service_id/2f9c36f2-86f5-4a6f-a853-b02f0d47d881 mysqld_exporter Starting push /agent_id/5e5056be-b333-4b9d-8db7-ba2ee1b477d6 /service_id/72c6b5ec-6630-436c-ad51-4bab5956fedf mysqld_exporter Starting push /agent_id/6025ba98-8bb5-4ea6-a7ea-8f7549e9bccf /service_id/1f4e9928-6fab-4ef7-b2b5-6585132d47ae mysqld_exporter Starting push /agent_id/64c816f6-d752-41e5-b1d7-f36e7739be2b /service_id/c2a3b4cb-c8ea-4792-bbe5-df9429480943 mysqld_exporter Starting push /agent_id/661132aa-4ed5-4283-aff3-b26031fc4d75 /service_id/7778cfb7-fbb6-46e9-81cf-de98d93baf28 mysqld_exporter Starting push /agent_id/66cfa45e-c96f-4c07-b520-f24f19dfaa2f /service_id/897b0e8f-2384-492e-94e2-4a31ebb298a7 mysqld_exporter Starting push /agent_id/6a1e4cc7-fba5-4664-be07-cc161cbe97ec /service_id/fff37369-e4d3-410f-acf0-aaee15da598f mysqld_exporter Starting push /agent_id/6aaf16c6-4946-4ed5-ae27-524703dcb495 /service_id/0c10629f-608c-4fbf-b4a5-4c8715d413b4 mysqld_exporter Starting push /agent_id/6b399490-05f0-44d9-8b5c-ef17dd4041e9 /service_id/53dea499-b2e8-4b18-bc33-1f8c8d831148 mysqld_exporter Starting push /agent_id/6d3f0a3f-6640-44a2-85e2-dc6221910c35 /service_id/bfb3ff2b-ae6f-4b1a-93fa-c358efd9e19a mysqld_exporter Starting push /agent_id/730ee723-7004-49e9-acab-eb3c1c2b2bee /service_id/474fcb86-656d-4194-8044-fbef2f2f4f10 mysqld_exporter Starting push /agent_id/776bda49-8e14-4024-98b1-034bc809c71c /service_id/a62a445c-495e-4f43-a8e2-715e9330c198 mysqld_exporter Starting push /agent_id/7882b691-2331-4156-9ead-65d8673a8633 /service_id/5f6db0f8-353d-4a15-adef-d9f807dea7c3 mysqld_exporter Starting push /agent_id/7eb80440-4d96-49ff-82c0-3c65a29c0135 /service_id/32f2b49d-169a-4a4f-b989-059ccda00762 mysqld_exporter Starting push /agent_id/823957d6-0225-43dc-bf6c-c4da57b38e62 /service_id/639b1c0c-dc95-4d77-ae0e-93cd16b854bd mysqld_exporter Starting push /agent_id/86a369c7-3802-4db9-bfaa-d076dcaa88fd /service_id/9241e6ff-2820-4951-b333-8ebb125534fa mysqld_exporter Starting push /agent_id/87bb9366-67d5-4c43-85e9-769a08695a1d /service_id/e66d4717-a610-49ba-a0ea-9b004c556fcf mysqld_exporter Starting push /agent_id/90dc52d2-8098-4e70-80b3-675ec6ea9f3e /service_id/668d4ec7-ceb6-49ab-bf76-393e511895a3 mysqld_exporter Starting push /agent_id/9294ae4a-b8f0-4c5c-9d5c-94a8ec06ddc4 /service_id/99c398e5-7446-4a23-abde-e6622e71854d mysqld_exporter Starting push /agent_id/93a64f4d-7d17-49ba-ac40-586619cc2681 /service_id/cdd2a1c2-b478-4e53-86ad-b483536c73f7 mysqld_exporter Starting push /agent_id/9650bbed-928f-4c6d-94a7-098c04b46bc4 /service_id/606b76c0-3034-49c7-9c9d-e164b3d970c9 mysqld_exporter Starting push /agent_id/96552b31-1607-4469-a1ab-c2e0a43da619 /service_id/1d6a98ef-6aee-4347-ba9a-138bb194899d mysqld_exporter Starting push /agent_id/96911375-e202-4d43-b2ef-09dc13a27618 /service_id/05731e68-b833-4968-945b-d05a8f3501c2 mysqld_exporter Starting push /agent_id/9a39d202-1bcc-45bd-a80c-efe3f010478a /service_id/3a975a8e-c00a-4015-900b-24517e76107d mysqld_exporter Starting push /agent_id/a12a374f-739b-4e2f-a220-ec934adb7d14 /service_id/707ad363-9758-441d-8d3d-52e786cd5866 mysqld_exporter Starting push /agent_id/a2a82728-5554-4c89-92f7-3e0c0b62df8b /service_id/c64020e5-113b-4db0-86cd-2b0244310e64 mysqld_exporter Starting push /agent_id/a69da6a6-8def-4295-ab57-3650da44129c /service_id/7e374ae7-2223-49e7-95c7-f07466d5beee mysqld_exporter Starting push /agent_id/b4245760-8206-4fac-a771-d82d594d6d9d /service_id/f55cbb87-46e8-4b1b-9b56-46c947194792 mysqld_exporter Starting push /agent_id/bbdda7af-11f9-4be4-a9a2-4ba649d44a1b /service_id/d0a3251e-953d-4126-b448-3ece45d499b3 mysqld_exporter Starting push /agent_id/bcb9b544-6b87-4126-ae15-a25604da4dab /service_id/abc4b946-184a-42bf-ae2a-b8e1bcfd95c8 mysqld_exporter Starting push /agent_id/be6f372e-496a-4c0f-8281-4af43987a3a4 /service_id/33b96340-1fc4-4b69-9079-224bc9f67279 mysqld_exporter Starting push /agent_id/c006af94-96bc-4b14-8f87-7fb53b51c457 /service_id/87717038-7bde-439d-b8a8-7bab080c0f8d mysqld_exporter Starting push /agent_id/c0dae363-da8e-45c0-9ae6-f19d6729a3e7 /service_id/37404044-8da1-466c-abdc-f19584525539 mysqld_exporter Starting push /agent_id/c460f0d7-822b-4ad9-9ed2-e6337e4e012d /service_id/3ca2e92b-7bc8-4556-9aac-3e06b93fb80c mysqld_exporter Starting push /agent_id/c5031bcb-02ee-449a-bf9a-efc555931972 /service_id/422c5a5f-9ae3-495b-92d7-afb5c2f79185 mysqld_exporter Starting push /agent_id/c53b6d73-9828-412d-9e81-d8ba1c1575fc /service_id/fa22cb45-05ce-48dc-bfd8-a2c741890ab5 mysqld_exporter Starting push /agent_id/c92ab360-4fd7-40b9-922e-2a3af8228c16 /service_id/8756603a-c419-449b-bcbc-7e07a32d9623 mysqld_exporter Starting push /agent_id/cd18d4ee-fa50-4b62-b063-0f021d735989 /service_id/e36377a6-822b-44c3-8af0-ce46b5afba88 mysqld_exporter Starting push /agent_id/d454c5ab-4b91-4bde-8607-e947e3d5d9f4 /service_id/4f49cbe8-c4bb-4086-a1df-0c25558af6e6 mysqld_exporter Starting push /agent_id/d46b5421-b41c-4ada-8d8c-a45fa1265828 /service_id/66041980-30a7-498a-802f-e3844a3aff55 mysqld_exporter Starting push /agent_id/daa4fa44-aeaa-4514-8294-b6875a424d97 /service_id/f549642a-a3a8-45a0-acdd-77c7ad789168 mysqld_exporter Starting push /agent_id/e2bd4e0f-9042-490c-9208-71c916b2378b /service_id/a0c1f671-f6fd-45ae-9b94-f91219f5ffdf mysqld_exporter Starting push /agent_id/e2dff880-18f2-41b9-a047-22d32a23e8eb /service_id/97063093-1320-4440-a84c-00bc263d2a23 mysqld_exporter Starting push /agent_id/e83cf0b4-8520-4a00-a86b-ee71bbb89422 /service_id/92a46c24-b5fa-4630-b1f7-eddf1b4a3817 mysqld_exporter Starting push /agent_id/e9f697d2-a1bd-48bd-9d66-5954d8c10e05 /service_id/d9116972-5a3b-4e6a-8326-6d29925fb8f2 mysqld_exporter Starting push /agent_id/eba9db74-9098-47dc-9a85-7b97878215ee /service_id/e9f1df77-7ada-4a0a-b5bd-0bcfb8f31b83 mysqld_exporter Starting push /agent_id/ec6d8b5f-795d-4142-9724-68ea107d526a /service_id/d2254f06-4865-4514-bc13-63562652bb3b mysqld_exporter Starting push /agent_id/eec7b426-1dfe-4379-8ccb-2fb56acecbc0 /service_id/1d028e90-b1cc-4a8e-8c1f-333202906e47 mysqld_exporter Starting push /agent_id/efc02063-1b1d-4daf-8582-bf0cfe2355a4 /service_id/f365144f-b398-4fe7-a33a-0434f5e0e271 mysqld_exporter Starting push /agent_id/f7e3cbab-ca34-4aa6-a57c-138a41ea2316 /service_id/692697b5-a6f1-41b3-9bf3-aa9606d7c6bb mysqld_exporter Starting push /agent_id/fae2c724-35e0-4dec-9486-69ab486cc218 /service_id/4dce80d4-05cd-46ee-b84a-07e8571f9d02 mysqld_exporter Starting push /agent_id/fbcd3cc2-ea54-4509-8582-1476b02e2767 /service_id/aa7e85d4-7b55-405e-89a0-023719c69c2f mysqld_exporter Starting push /agent_id/feb9d63f-031c-4151-afb4-3b22a8f66a1a /service_id/805779f3-08ee-4348-89d1-5680bf1fecaf mongodb_exporter Starting push /agent_id/15eb5382-0ce8-4f10-9586-77d21e267507 /service_id/22dab00a-4d1a-42ba-9ec1-19dbed3667a6 mongodb_exporter Starting push /agent_id/9a5582e2-79a6-46ef-b8bc-f9d82d5caf8a /service_id/78268dad-a48b-45d5-af9b-fc9eb5a6c82a mongodb_exporter Starting push /agent_id/b2d520a8-9074-4e2d-ab01-5e2cd4bfc2ec /service_id/964e8e2e-6d1f-4576-bd4e-7cd342d79c3f mysql_perfschema_agent Starting /agent_id/017bfa92-b6d1-4e2f-882a-2396df2726c1 /service_id/33b96340-1fc4-4b69-9079-224bc9f67279 mysql_perfschema_agent Starting /agent_id/069d3a54-05ab-47dd-9c46-c5847d352cbf /service_id/f0150ba2-9a16-4fe0-a086-ae43da7a6804 mysql_perfschema_agent Starting /agent_id/0771dc67-da70-4a6e-8c8b-c1a7815b41ea /service_id/246a40b0-e539-4e8d-853f-7498cdbc83cc mysql_perfschema_agent Starting /agent_id/082c037d-53d0-467d-8ff9-edf8e7bb8253 /service_id/fff37369-e4d3-410f-acf0-aaee15da598f mysql_perfschema_agent Starting /agent_id/0fb1104c-ec0f-4c69-9388-eeea03e0df79 /service_id/8756603a-c419-449b-bcbc-7e07a32d9623 mysql_perfschema_agent Starting /agent_id/1088eea7-0be1-44f9-b330-908501cf3c8e /service_id/c4825858-e7be-4714-928d-5d4ba567bbd2 mysql_perfschema_agent Starting /agent_id/10e78439-f761-4230-877b-6207fa92d471 /service_id/2e92728d-3426-4948-b9e4-3ad2d0fbd865 mysql_perfschema_agent Starting /agent_id/1429e51d-922d-4473-a760-12fa1b1fa915 /service_id/d8a151a6-ef8c-488a-88d9-abf0b2f88996 mysql_perfschema_agent Starting /agent_id/15cf3faa-a385-42fc-9a5e-9f86e6dace66 /service_id/e36377a6-822b-44c3-8af0-ce46b5afba88 mysql_perfschema_agent Starting /agent_id/191982cc-6e15-4dce-bf4d-1bbdc56aa051 /service_id/422c5a5f-9ae3-495b-92d7-afb5c2f79185 mysql_perfschema_agent Starting /agent_id/1ba493c9-cc1a-40a8-baeb-2151fdf7f3ba /service_id/f918fd4c-10a1-4110-b829-f0f789eda5ba mysql_perfschema_agent Starting /agent_id/1f96ff2a-6130-4823-a976-d6f4a60923e9 /service_id/2c5bfbf3-a68f-4953-84e3-968abbf13c67 mysql_perfschema_agent Starting /agent_id/272ee3de-d9d7-4ffb-b718-ba401f1257ac /service_id/aa7e85d4-7b55-405e-89a0-023719c69c2f mysql_perfschema_agent Starting /agent_id/29d0b146-8dc3-4c2f-97f5-8b943987ae7d /service_id/66041980-30a7-498a-802f-e3844a3aff55 mysql_perfschema_agent Starting /agent_id/2a16d822-34cc-4044-b0b0-12c9435622ba /service_id/0c10629f-608c-4fbf-b4a5-4c8715d413b4 mysql_perfschema_agent Starting /agent_id/31334c22-2d1b-4d60-abe7-cec11798a030 /service_id/2f9c36f2-86f5-4a6f-a853-b02f0d47d881 mysql_perfschema_agent Starting /agent_id/3261bc86-f703-4a92-ba67-df98ee776ac6 /service_id/707ad363-9758-441d-8d3d-52e786cd5866 mysql_perfschema_agent Starting /agent_id/33df2684-16c0-46a4-8602-aa9587329674 /service_id/5107dd70-dbd5-4e8b-b0ec-22370349045e mysql_perfschema_agent Unknown /agent_id/37282a4c-4f5a-4f67-9f05-25ec7bd29d9e /service_id/9241e6ff-2820-4951-b333-8ebb125534fa mysql_perfschema_agent Starting /agent_id/3a3fab7e-d0f0-47e5-ae6a-3766de2a5a47 /service_id/3ea495ba-5453-457e-9852-b5beb2340f91 mysql_perfschema_agent Starting /agent_id/3be0d58e-39d5-44ad-b6a7-55b5b7e0bb88 /service_id/897b0e8f-2384-492e-94e2-4a31ebb298a7 mysql_perfschema_agent Starting /agent_id/3f330fe3-1667-4dc9-b197-352eb4eb8a61 /service_id/7e374ae7-2223-49e7-95c7-f07466d5beee mysql_perfschema_agent Starting /agent_id/40676232-b021-4bdf-a028-702fd11c8505 /service_id/1d028e90-b1cc-4a8e-8c1f-333202906e47 mysql_perfschema_agent Starting /agent_id/411af973-5c22-47de-9283-1d2bd4f08788 /service_id/fa22cb45-05ce-48dc-bfd8-a2c741890ab5 mysql_perfschema_agent Unknown /agent_id/415c5871-8e40-492e-a42d-dedbe9fae0b7 /service_id/92a46c24-b5fa-4630-b1f7-eddf1b4a3817 mysql_perfschema_agent Unknown /agent_id/43fed475-0d09-47f4-83e6-c28e19344a99 /service_id/21927fa6-161a-429c-8be3-6f5b1d4f29dc mysql_perfschema_agent Unknown /agent_id/50219b70-a070-4400-9899-30fd80f385fd /service_id/668d4ec7-ceb6-49ab-bf76-393e511895a3 mysql_perfschema_agent Unknown /agent_id/50f499a6-6fe0-4ab1-87cb-c7df967ae661 /service_id/f365144f-b398-4fe7-a33a-0434f5e0e271 mysql_perfschema_agent Unknown /agent_id/5ceffb74-3b4f-4d54-b2e3-7422fb42501b /service_id/e863e544-1e68-40ef-a945-1bf594b10cee mysql_perfschema_agent Unknown /agent_id/5dc4ff5b-1b2a-4576-aadb-28005336f868 /service_id/a62a445c-495e-4f43-a8e2-715e9330c198 mysql_perfschema_agent Unknown /agent_id/63abe6bd-6faa-4773-af27-b37f9e1105aa /service_id/3e58842a-ce49-4d3b-b187-5e1d407c3d3c mysql_perfschema_agent Starting /agent_id/647589f0-0038-4483-8c19-d4317c3a32cf /service_id/0188b0a8-d233-4dfb-a0bc-57917fea84d8 mysql_perfschema_agent Unknown /agent_id/6689e9c6-64b2-4a06-b93d-1372fc0bb824 /service_id/606b76c0-3034-49c7-9c9d-e164b3d970c9 mysql_perfschema_agent Unknown /agent_id/68cc3101-4c96-4c80-b36f-145d6ff5c933 /service_id/f549642a-a3a8-45a0-acdd-77c7ad789168 mysql_perfschema_agent Unknown /agent_id/6fd3be26-f4f6-44ec-b946-4d07b6cf289f /service_id/e66d4717-a610-49ba-a0ea-9b004c556fcf mysql_perfschema_agent Unknown /agent_id/703084ef-4a66-4bd9-9652-db8569b797f8 /service_id/97063093-1320-4440-a84c-00bc263d2a23 mysql_perfschema_agent Unknown /agent_id/71b1cae1-d305-4e66-8880-326200a22f3e /service_id/1737d3fd-6bba-48e5-ad96-955ce3569644 mysql_perfschema_agent Unknown /agent_id/75fdd4dd-bab2-4c2e-962a-62490262bbb5 /service_id/3a975a8e-c00a-4015-900b-24517e76107d mysql_perfschema_agent Unknown /agent_id/762420cd-9d7d-4817-853b-a2aebfbfe632 /service_id/32f2b49d-169a-4a4f-b989-059ccda00762 mysql_perfschema_agent Unknown /agent_id/76ca33bc-ec81-400c-a395-fe1e378bc240 /service_id/b70a3649-a813-428e-b266-8044b8048395 mysql_perfschema_agent Unknown /agent_id/77cbd99b-c30d-4fd3-86a1-2f6e05025b44 /service_id/d0a3251e-953d-4126-b448-3ece45d499b3 mysql_perfschema_agent Starting /agent_id/7a0d5a78-e7bd-475d-aa60-be8b8717b5c3 /service_id/805779f3-08ee-4348-89d1-5680bf1fecaf mysql_perfschema_agent Starting /agent_id/7b396c32-6561-43d1-a3af-e82aa424c1d2 /service_id/c2a3b4cb-c8ea-4792-bbe5-df9429480943 mysql_perfschema_agent Starting /agent_id/7b99128d-c539-4f11-bfb3-09274daf159b /service_id/05731e68-b833-4968-945b-d05a8f3501c2 mysql_perfschema_agent Starting /agent_id/7b9b62fa-0062-4373-85f8-1e16e71eea7a /service_id/e9f1df77-7ada-4a0a-b5bd-0bcfb8f31b83 mysql_perfschema_agent Starting /agent_id/7de719e9-d1f9-498b-8e6f-cbe5be58ba7e /service_id/00130b9a-e0c9-4b88-b13f-67eef9da6fa9 mysql_perfschema_agent Starting /agent_id/84c58c90-5797-4dbd-b574-74e26f93bede /service_id/f4b458e0-b174-4ec6-be55-860ec4b9c0a7 mysql_perfschema_agent Starting /agent_id/8ec573e6-1dca-4c37-9d1d-a6a30f0bd604 /service_id/df80b5d3-fd1c-42f4-86c8-ac89f98c355d mysql_perfschema_agent Starting /agent_id/915f9d9d-5252-4872-9ce5-dcf36835e5e3 /service_id/72c6b5ec-6630-436c-ad51-4bab5956fedf mysql_perfschema_agent Starting /agent_id/98355db5-6bb6-4ef2-961b-57fd520fac92 /service_id/99c398e5-7446-4a23-abde-e6622e71854d mysql_perfschema_agent Starting /agent_id/9a67ae0d-3742-4096-8a7f-2c45b39fdc81 /service_id/3ca2e92b-7bc8-4556-9aac-3e06b93fb80c mysql_perfschema_agent Starting /agent_id/9cd8d21d-a758-4e3a-b407-8e248ff17aff /service_id/53dea499-b2e8-4b18-bc33-1f8c8d831148 mysql_perfschema_agent Starting /agent_id/a30352e9-035d-40f1-b094-6729a82c7b81 /service_id/abc4b946-184a-42bf-ae2a-b8e1bcfd95c8 mysql_perfschema_agent Starting /agent_id/a5226004-712b-4fb0-8c4f-e1a2c9ea509f /service_id/037194d2-2093-47de-9344-82a460992a48 mysql_perfschema_agent Starting /agent_id/a58a3fef-f44d-4ed9-a0ef-63774fe7965a /service_id/5f6db0f8-353d-4a15-adef-d9f807dea7c3 mysql_perfschema_agent Starting /agent_id/b08a24bc-4a60-4886-a5a7-9627e56d6f9d /service_id/bfb3ff2b-ae6f-4b1a-93fa-c358efd9e19a mysql_perfschema_agent Starting /agent_id/b1710b25-eca6-42b9-9c92-a82e8b2cb0df /service_id/7778cfb7-fbb6-46e9-81cf-de98d93baf28 mysql_perfschema_agent Starting /agent_id/b2d6b5a6-77a0-4d14-870e-f31e1e755969 /service_id/639b1c0c-dc95-4d77-ae0e-93cd16b854bd mysql_perfschema_agent Starting /agent_id/b4dabcd8-603a-4869-8ec7-f443df88a019 /service_id/646a0d5b-4f6a-4a00-88be-d4ed0955b3b6 mysql_perfschema_agent Starting /agent_id/b5d71981-3cb8-4b88-b13c-98a0f66eb47b /service_id/474fcb86-656d-4194-8044-fbef2f2f4f10 mysql_perfschema_agent Starting /agent_id/b700b696-1653-4c86-8f71-b849293024b9 /service_id/16f9c61b-0f6c-488a-b5a8-06f94e7e558c mysql_perfschema_agent Starting /agent_id/b70541fe-13e9-4562-9dce-e3b9c97efd08 /service_id/1d6a98ef-6aee-4347-ba9a-138bb194899d mysql_perfschema_agent Starting /agent_id/b8a44986-5416-41a2-9ea0-d873782b29cd /service_id/2470d3ef-c583-457f-9996-1bc6953b6956 mysql_perfschema_agent Starting /agent_id/b8fa1221-5289-4cc5-af1e-2092d0fd7663 /service_id/f55cbb87-46e8-4b1b-9b56-46c947194792 mysql_perfschema_agent Starting /agent_id/bdc7fab7-b8c6-4236-8b87-1e61c5cda7dc /service_id/9a3aba6c-a9d6-4c6f-9941-76709e90a241 mysql_perfschema_agent Starting /agent_id/be91cb02-32c4-4361-8874-9777673afd08 /service_id/692697b5-a6f1-41b3-9bf3-aa9606d7c6bb mysql_perfschema_agent Starting /agent_id/c468c268-b245-4aac-8fbe-915f328576b8 /service_id/83be5d84-e34e-4e5c-ab21-b4261431eba7 mysql_perfschema_agent Starting /agent_id/c57dea80-7771-4454-a604-e7c3a2067b75 /service_id/2d27d91a-71f0-4751-9a66-e5533cfe6795 mysql_perfschema_agent Starting /agent_id/cf67de2f-4786-4a2b-bfde-e5cc236adec4 /service_id/235665cf-6155-4098-b77b-8b301240df14 mysql_perfschema_agent Starting /agent_id/d0b61cfa-904c-4d62-9b91-a891d6b05f94 /service_id/87717038-7bde-439d-b8a8-7bab080c0f8d mysql_perfschema_agent Starting /agent_id/d86517bc-fead-43cb-80cf-7f15f1438944 /service_id/c64020e5-113b-4db0-86cd-2b0244310e64 mysql_perfschema_agent Unknown /agent_id/d8c137cd-ccd6-4120-b5ec-ead67b92f25c /service_id/1db5dd10-8d96-451d-9a26-6969871f93ad mysql_perfschema_agent Starting /agent_id/dc8763b4-5a6d-4769-a117-d31bf47cfec3 /service_id/1f4e9928-6fab-4ef7-b2b5-6585132d47ae mysql_perfschema_agent Starting /agent_id/dd302dae-cc09-4ddd-b738-17c7e0817db5 /service_id/37404044-8da1-466c-abdc-f19584525539 mysql_perfschema_agent Starting /agent_id/df9b42c0-7f14-463c-a30c-f6bb81bdf84b /service_id/5feea6cd-5d32-458f-9ce3-350cdc2e8f25 mysql_perfschema_agent Starting /agent_id/e47f38bb-7057-48ba-8bad-2a5024b1c0b5 /service_id/cdd2a1c2-b478-4e53-86ad-b483536c73f7 mysql_perfschema_agent Starting /agent_id/e4cb8e79-12ca-4d26-8d52-44a38b434e55 /service_id/a0c1f671-f6fd-45ae-9b94-f91219f5ffdf mysql_perfschema_agent Starting /agent_id/ea38610f-ef14-4327-ae0c-cd5a74606426 /service_id/4f49cbe8-c4bb-4086-a1df-0c25558af6e6 mysql_perfschema_agent Starting /agent_id/f15b37f9-d81c-494f-8bbf-3a9a14207ca0 /service_id/d9116972-5a3b-4e6a-8326-6d29925fb8f2 mysql_perfschema_agent Starting /agent_id/f242b266-5860-41d8-842c-42220f81d8cd /service_id/4dce80d4-05cd-46ee-b84a-07e8571f9d02 mysql_perfschema_agent Starting /agent_id/fa71ba64-baa1-454a-ad59-9440da021ee1 /service_id/621cef6e-0283-45f4-8f1a-e6a1c161f441 mysql_perfschema_agent Starting /agent_id/fd72a6c0-a25c-498e-93dc-2b8dcb4fc6dd /service_id/d2254f06-4865-4514-bc13-63562652bb3b mongodb_profiler_agent Starting /agent_id/b93ae55a-6c43-4726-8114-241f3a3ae924 /service_id/78268dad-a48b-45d5-af9b-fc9eb5a6c82a mongodb_profiler_agent Starting /agent_id/d7ebc526-51d6-4eed-95f0-b0f518e6553b /service_id/964e8e2e-6d1f-4576-bd4e-7cd342d79c3f mongodb_profiler_agent Starting /agent_id/e8df4b51-5348-4b88-9443-65ad2e98e0f4 /service_id/22dab00a-4d1a-42ba-9ec1-19dbed3667a6 vmagent Running push /agent_id/68edde90-07bd-42c4-97f3-6f0c4f691ffbAdditional info:
Metrics are collected after upgrade even when agents are in Starting state
When pmm-client is also upgraded to 2.21.0 all agents are Running again