Registering node with Grafana Admin flag enabled and non-admin role doesn't work

General

Escalation

General

Escalation

Description

So in MS we force-readd pmm-agent with each upgrade as part of our automation stack, it's an easy way to setup agents without having to care whether it's a new installation or it's an upgrade

This issue happens when we upgraded from 2.34.0 to 2.35.0. pmm-agents would fail with a 5xx, logs indicating authentication issue - however, the same credentials used between web UI, API, and pmm-agents were verified to be correct and not locked/timed out by Grafana bruteforce protection

Issue does not happen when we switched the 2.35.0 server container back to 2.34.0, removed all 2.35.0 clients, reinstalled 2.34.0 clients then readded them with --force

Also, this issue can be replicated on my local test environment that was running 2.34.0

PMM client log

$ pmm-agent setup --force @/etc/rdba/pmm/pmm-agent-flags.txt
INFO[2023-02-23T13:29:29.054+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter  component=setup
INFO[2023-02-23T13:29:29.054+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter  component=setup
INFO[2023-02-23T13:29:29.054+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter  component=setup
INFO[2023-02-23T13:29:29.054+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter  component=setup
INFO[2023-02-23T13:29:29.055+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter  component=setup
INFO[2023-02-23T13:29:29.055+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter  component=setup
INFO[2023-02-23T13:29:29.055+00:00] Using /usr/local/percona/pmm2/exporters/azure_exporter  component=setup
INFO[2023-02-23T13:29:29.055+00:00] Using /usr/local/percona/pmm2/exporters/vmagent  component=setup
Checking local pmm-agent status...
DEBU[2023-02-23T13:59:04.977+00:00] POST /local/Status HTTP/1.1
Host: 127.0.0.1:7777
User-Agent: Go-http-client/1.1
Content-Length: 3
Accept: application/json
Content-Type: application/json
Accept-Encoding: gzip

{}
  component=setup
DEBU[2023-02-23T13:59:04.978+00:00] HTTP/1.1 200 OK
Content-Length: 241
Content-Type: application/json
Date: Thu, 23 Feb 2023 13:59:04 GMT
Grpc-Metadata-Content-Type: application/grpc

{
  "agent_id":  "",
  "runs_on_node_id":  "",
  "server_info":  null,
  "agents_info":  [],
  "config_filepath":  "/usr/local/percona/pmm2/config/pmm-agent.yaml",
  "agent_version":  "2.35.0",
  "node_name":  "",
  "connection_uptime":  0
}  component=setup
DEBU[2023-02-23T13:59:04.978+00:00] Status error: <nil>                           component=setup
pmm-agent is running.
Registering pmm-agent on PMM Server...
DEBU[2023-02-23T13:59:04.979+00:00] POST /v1/management/Node/Register HTTP/1.1
Host: 10.30.19.22:8443
User-Agent: Go-http-client/1.1
Content-Length: 226
Accept: application/json
Authorization: Basic XXXXXXXXXXXXXXXXXXXXXXXX
Content-Type: application/json
Accept-Encoding: gzip

{"node_type":"GENERIC_NODE","node_name":"platform-team04","address":"10.30.19.23","machine_id":"/machine_id/10eeea8a634748afb699186b69f080c2","distro":"linux","reregister":true,"metrics_mode":"PUSH","disable_collectors":null}
  component=setup
DEBU[2023-02-23T13:59:05.073+00:00] HTTP/1.1 500 Internal Server Error
Content-Length: 95
Connection: keep-alive
Content-Type: application/json
Date: Thu, 23 Feb 2023 13:59:05 GMT
Server: nginx
Strict-Transport-Security: max-age=63072000; includeSubdomains;

{
  "error":  "Internal server error.",
  "code":  13,
  "message":  "Internal server error."
}  component=setup
DEBU[2023-02-23T13:59:05.073+00:00] Register error: &node.RegisterNodeDefault{_statusCode:500, Payload:(*node.RegisterNodeDefaultBody)(0xc0000b2960)}  component=setup
Failed to register pmm-agent on PMM Server: Internal server error..

pmm-managed log on PMM server

INFO[2023-02-23T13:25:21.392+00:00] Starting Stream /agent.Agent/Connect ...      agent_id=/agent_id/413bc08c-292b-45d6-9ff8-f22130d4b387 request=85b5e825-b37d-11ed-9a96-525400a9\
1930
WARN[2023-02-23T13:25:21.394+00:00] Failed to authenticate connected pmm-agent &{ID:/agent_id/413bc08c-292b-45d6-9ff8-f22130d4b387 Version:2.35.0 MetricsPort:0}.  agent_id=/agent\
_id/413bc08c-292b-45d6-9ff8-f22130d4b387 request=85b5e825-b37d-11ed-9a96-525400a91930
WARN[2023-02-23T13:25:21.395+00:00] Stream /agent.Agent/Connect done in 2.142882ms with gRPC error: rpc error: code = PermissionDenied desc = No Agent with ID "/agent_id/413bc08c\
-292b-45d6-9ff8-f22130d4b387".  agent_id=/agent_id/413bc08c-292b-45d6-9ff8-f22130d4b387 request=85b5e825-b37d-11ed-9a96-525400a91930
INFO[2023-02-23T13:25:21.687+00:00] Starting RPC /management.Node/RegisterNode ...  request=85e2d7f1-b37d-11ed-9a96-525400a91930
ERRO[2023-02-23T13:25:21.734+00:00] RPC /management.Node/RegisterNode done in 47.259216ms with unexpected error: clientError: POST http://127.0.0.1:3000/api/auth/keys -> 403 {"ac\
cessErrorId":"ACE5271461526","message":"You'll need additional permissions to perform this action. Permissions needed: apikeys:create","title":"Access denied"}

github.com/percona/pmm/managed/services/grafana.(*Client).do
        /home/builder/rpm/BUILD/pmm-5e80fd1f3fdddbae98002e819e9396043b7a9740/src/github.com/percona/pmm/managed/services/grafana/client.go:149
github.com/percona/pmm/managed/services/grafana.(*Client).createAPIKey
        /home/builder/rpm/BUILD/pmm-5e80fd1f3fdddbae98002e819e9396043b7a9740/src/github.com/percona/pmm/managed/services/grafana/client.go:559
github.com/percona/pmm/managed/services/grafana.(*Client).CreateAdminAPIKey
        /home/builder/rpm/BUILD/pmm-5e80fd1f3fdddbae98002e819e9396043b7a9740/src/github.com/percona/pmm/managed/services/grafana/client.go:376
github.com/percona/pmm/managed/services/management.(*NodeService).Register
        /home/builder/rpm/BUILD/pmm-5e80fd1f3fdddbae98002e819e9396043b7a9740/src/github.com/percona/pmm/managed/services/management/node.go:140
github.com/percona/pmm/managed/services/management/grpc.(*nodeServer).RegisterNode
        /home/builder/rpm/BUILD/pmm-5e80fd1f3fdddbae98002e819e9396043b7a9740/src/github.com/percona/pmm/managed/services/management/grpc/node_server.go:39
github.com/percona/pmm/api/managementpb._Node_RegisterNode_Handler.func1
        /home/builder/rpm/BUILD/pmm-5e80fd1f3fdddbae98002e819e9396043b7a9740/src/github.com/percona/pmm/api/managementpb/node_grpc.pb.go:88
github.com/grpc-ecosystem/go-grpc-middleware/validator.UnaryServerInterceptor.func1
        /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/validator/validator.go:47
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/percona/pmm/managed/utils/interceptors.UnaryServiceEnabledInterceptor.func1
        /home/builder/rpm/BUILD/pmm-5e80fd1f3fdddbae98002e819e9396043b7a9740/src/github.com/percona/pmm/managed/utils/interceptors/service.go:39
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).UnaryServerInterceptor.func1
        /home/builder/go/pkg/mod/github.com/!percona-!lab/go-grpc-prometheus@v0.0.0-20230105215234-10537622c253/server_metrics.go:114
github.com/percona/pmm/managed/utils/interceptors.Unary.func1.1
        /home/builder/rpm/BUILD/pmm-5e80fd1f3fdddbae98002e819e9396043b7a9740/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:106
github.com/percona/pmm/managed/utils/interceptors.logRequest
        /home/builder/rpm/BUILD/pmm-5e80fd1f3fdddbae98002e819e9396043b7a9740/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:78
github.com/percona/pmm/managed/utils/interceptors.Unary.func1
        /home/builder/rpm/BUILD/pmm-5e80fd1f3fdddbae98002e819e9396043b7a9740/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:104
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1