Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

gRPC error: rpc error: code = PermissionDenied desc = No Agent with ID

Description

Inside PMM server container, check the pmm-managed.log file and we can see the below error details.

time="2024-11-05T16:21:49.133+00:00" level=warning msg="Failed to authenticate connected pmm-agent &{ID:/agent_id/cd4348d7-3454-47a5-bb32-7b77ecf9e987 Version:2.41.2 MetricsPort:0}." agent_id=/agent_id/cd4348d7-3454-47a5-bb32-7b77ecf9e987 request=0f058ba9-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:21:49.133+00:00" level=warning msg="Stream /agent.Agent/Connect done in 804.292µs with gRPC error: rpc error: code = PermissionDenied desc = No Agent with ID \"/agent_id/cd4348d7-3454-47a5-bb32-7b77ecf9e987\"." agent_id=/agent_id/cd4348d7-3454-47a5-bb32-7b77ecf9e987 request=0f058ba9-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:21:50.425+00:00" level=info msg="Starting RPC /server.Server/Readiness ..." request=0fcac9cc-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:21:50.427+00:00" level=info msg="RPC /server.Server/Readiness done in 1.696981ms." request=0fcac9cc-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:21:51.466+00:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2024-11-05T16:21:51.560+00:00" level=info msg="SetState response: ." agent_id=/agent_id/285f52a8-714f-4eae-8946-bbbf3bf913b1 request=7f113689-86d2-11ef-aef0-0242ac110002
time="2024-11-05T16:21:53.034+00:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2024-11-05T16:21:53.038+00:00" level=info msg="SetState response: ." agent_id=/agent_id/0ab837ec-3c77-4ff4-bb7c-2902be7642e9 request=7afa2e5b-86d2-11ef-aef0-0242ac110002
time="2024-11-05T16:21:53.481+00:00" level=info msg="Starting RPC /server.Server/Readiness ..." request=119d2cc6-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:21:53.484+00:00" level=info msg="RPC /server.Server/Readiness done in 2.996145ms." request=119d2cc6-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:21:56.540+00:00" level=info msg="Starting RPC /server.Server/Readiness ..." request=136fe184-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:21:56.542+00:00" level=info msg="RPC /server.Server/Readiness done in 1.762801ms." request=136fe184-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:21:56.828+00:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2024-11-05T16:21:56.872+00:00" level=info msg="SetState response: ." agent_id=/agent_id/e6f63601-ffe9-4017-bcf6-0cde4f38e3a8 request=82a9c7b1-86d2-11ef-aef0-0242ac110002
time="2024-11-05T16:21:57.548+00:00" level=info msg="Starting RPC /inventory.Services/ListActiveServiceTypes ..." request=1409bcad-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:21:57.549+00:00" level=info msg="RPC /inventory.Services/ListActiveServiceTypes done in 496.139µs." request=1409bcad-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:21:57.550+00:00" level=info msg="Starting RPC /inventory.Services/ListActiveServiceTypes ..." request=1409fc6c-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:21:57.550+00:00" level=info msg="RPC /inventory.Services/ListActiveServiceTypes done in 414.251µs." request=1409fc6c-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:21:57.552+00:00" level=info msg="Starting RPC /inventory.Services/ListActiveServiceTypes ..." request=140a57d3-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:21:57.553+00:00" level=info msg="RPC /inventory.Services/ListActiveServiceTypes done in 396.813µs." request=140a57d3-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:21:59.600+00:00" level=info msg="Starting RPC /server.Server/Readiness ..." request=1542bc52-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:21:59.601+00:00" level=info msg="RPC /server.Server/Readiness done in 1.505371ms." request=1542bc52-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:01.182+00:00" level=info msg="SetState response: ." agent_id=/agent_id/7c12418b-b867-4f4f-bbe4-3be5e7125627 request=7fa5e7c1-86d2-11ef-aef0-0242ac110002
time="2024-11-05T16:22:01.184+00:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2024-11-05T16:22:02.344+00:00" level=info msg="SetState response: ." agent_id=/agent_id/7c12418b-b867-4f4f-bbe4-3be5e7125627 request=7fa5e7c1-86d2-11ef-aef0-0242ac110002
time="2024-11-05T16:22:02.351+00:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2024-11-05T16:22:02.655+00:00" level=info msg="Starting RPC /server.Server/Readiness ..." request=1714f9bc-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:02.657+00:00" level=info msg="RPC /server.Server/Readiness done in 1.652051ms." request=1714f9bc-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:03.562+00:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2024-11-05T16:22:03.602+00:00" level=info msg="SetState response: ." agent_id=/agent_id/bbdfdc1d-8cd4-4dd1-bed1-d8fcf0532712 request=05d04183-8aff-11ef-aef0-0242ac110002
time="2024-11-05T16:22:04.676+00:00" level=info msg="SetState response: ." agent_id=/agent_id/e15c5697-c980-42d7-9f3b-a04a333e7967 request=80f6ff71-86d2-11ef-aef0-0242ac110002
time="2024-11-05T16:22:04.683+00:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2024-11-05T16:22:05.476+00:00" level=info msg="Starting Stream /agent.Agent/Connect ..." agent_id=/agent_id/cd4348d7-3454-47a5-bb32-7b77ecf9e987 request=18c37184-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:05.477+00:00" level=warning msg="Failed to authenticate connected pmm-agent &{ID:/agent_id/cd4348d7-3454-47a5-bb32-7b77ecf9e987 Version:2.41.2 MetricsPort:0}." agent_id=/agent_id/cd4348d7-3454-47a5-bb32-7b77ecf9e987 request=18c37184-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:05.477+00:00" level=warning msg="Stream /agent.Agent/Connect done in 777.697µs with gRPC error: rpc error: code = PermissionDenied desc = No Agent with ID \"/agent_id/cd4348d7-3454-47a5-bb32-7b77ecf9e987\"." agent_id=/agent_id/cd4348d7-3454-47a5-bb32-7b77ecf9e987 request=18c37184-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:05.716+00:00" level=info msg="Starting RPC /server.Server/Readiness ..." request=18e809c0-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:05.717+00:00" level=info msg="RPC /server.Server/Readiness done in 1.442672ms." request=18e809c0-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:05.981+00:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2024-11-05T16:22:06.101+00:00" level=info msg="SetState response: ." agent_id=/agent_id/285f52a8-714f-4eae-8946-bbbf3bf913b1 request=7f113689-86d2-11ef-aef0-0242ac110002
time="2024-11-05T16:22:06.980+00:00" level=info msg="Starting RPC /inventory.Services/ListActiveServiceTypes ..." request=19a8eca8-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:06.981+00:00" level=info msg="RPC /inventory.Services/ListActiveServiceTypes done in 502.637µs." request=19a8eca8-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:08.670+00:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2024-11-05T16:22:08.726+00:00" level=info msg="SetState response: ." agent_id=/agent_id/f6eaf0fc-9176-4960-9658-9a3896a40144 request=4637a008-923f-11ef-aef0-0242ac110002
time="2024-11-05T16:22:08.783+00:00" level=info msg="Starting RPC /server.Server/Readiness ..." request=1abc13a5-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:08.785+00:00" level=info msg="RPC /server.Server/Readiness done in 1.708377ms." request=1abc13a5-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:11.294+00:00" level=warning msg="No explicit rule, falling back to Grafana admin." component=grafana/auth req="POST /v1/ui-events/Store"
time="2024-11-05T16:22:11.299+00:00" level=info msg="Starting RPC /uievents.UIEvents/Store ..." request=1c3bf5cc-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:11.299+00:00" level=info msg="RPC /uievents.UIEvents/Store done in 53.682µs." request=1c3bf5cc-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:11.859+00:00" level=info msg="Starting RPC /server.Server/Readiness ..." request=1c91583f-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:11.861+00:00" level=info msg="RPC /server.Server/Readiness done in 1.729143ms." request=1c91583f-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:13.005+00:00" level=info msg="SetState response: ." agent_id=/agent_id/138cd09b-69c2-45f8-ad63-360affc3236c request=7e25abbb-86d2-11ef-aef0-0242ac110002
time="2024-11-05T16:22:13.046+00:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2024-11-05T16:22:14.913+00:00" level=info msg="Starting RPC /server.Server/Readiness ..." request=1e635288-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:14.914+00:00" level=info msg="RPC /server.Server/Readiness done in 1.758021ms." request=1e635288-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:17.439+00:00" level=info msg="Starting Stream /agent.Agent/Connect ..." agent_id=/agent_id/cd4348d7-3454-47a5-bb32-7b77ecf9e987 request=1fe4d7ba-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:17.440+00:00" level=warning msg="Failed to authenticate connected pmm-agent &{ID:/agent_id/cd4348d7-3454-47a5-bb32-7b77ecf9e987 Version:2.41.2 MetricsPort:0}." agent_id=/agent_id/cd4348d7-3454-47a5-bb32-7b77ecf9e987 request=1fe4d7ba-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:17.440+00:00" level=warning msg="Stream /agent.Agent/Connect done in 779.024µs with gRPC error: rpc error: code = PermissionDenied desc = No Agent with ID \"/agent_id/cd4348d7-3454-47a5-bb32-7b77ecf9e987\"." agent_id=/agent_id/cd4348d7-3454-47a5-bb32-7b77ecf9e987 request=1fe4d7ba-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:17.977+00:00" level=info msg="Starting RPC /server.Server/Readiness ..." request=2036fbb4-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:17.979+00:00" level=info msg="RPC /server.Server/Readiness done in 1.375624ms." request=2036fbb4-9b92-11ef-aef0-0242ac110002
time="2024-11-05T16:22:18.067+00:00" level=info msg="Configuration reloaded." component=victoriametrics
time="2024-11-05T16:22:18.104+00:00" level=info msg="SetState response: ." agent_id=/agent_id/4d68ee98-1c77-4fa4-ae10-53ebb148b6f8 request=099d0991-923f-11ef-aef0-0242ac110002
time="2024-11-05T16:22:19.705+00:00" level=info msg=Starting... component=logs d=1.427e-06 request=logs

How to test

Inside PMM server container, check the pmm-managed.log file.

How to document

None

Activity

Show:

Naresh November 6, 2024 at 3:01 AM

Got it, thanks for the help.

Please close this ticket.

Nurlan Moldomurov November 5, 2024 at 4:43 PM

pmm-agent not registered on PMM Server trying to connect and that’s why failing. Please find which pmm-agent uses that agent id and try to re-register

Not a Bug

Details

Assignee

Reporter

Priority

Labels

Needs QA

Yes

Needs Doc

Yes

Affects versions

Smart Checklist

Created November 5, 2024 at 4:31 PM
Updated November 6, 2024 at 6:42 AM
Resolved November 6, 2024 at 6:42 AM
Configure