Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

grpc: received message larger than max

Description

Seeing this error every now and then in pmm-managed.log:

ERRO[2022-05-16T11:09:04.928+00:00] rpc error: code = ResourceExhausted desc = grpc: received message larger than max (33565605 vs. 20971520) failed to sent CollectRequest to QAN github.com/percona/pmm-managed/services/qan.(*Client).Collect /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/services/qan/client.go:252 github.com/percona/pmm-managed/services/agents.(*Handler).Run.func3 /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/services/agents/handler.go:135 runtime/pprof.Do /usr/local/go/src/runtime/pprof/runtime.go:40 github.com/percona/pmm-managed/services/agents.(*Handler).Run /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/services/agents/handler.go:134 github.com/percona/pmm-managed/services/agents/grpc.(*agentServer).Connect /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/services/agents/grpc/agent_server.go:41 github.com/percona/pmm/api/agentpb._Agent_Connect_Handler /home/builder/go/pkg/mod/github.com/percona/pmm@v0.0.0-20220404174627-9de9a91ebd60/api/agentpb/agent_grpc.pb.go:98 github.com/grpc-ecosystem/go-grpc-middleware/validator.StreamServerInterceptor.func1 /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/validator/validator.go:72 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1 /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 github.com/percona/pmm-managed/utils/interceptors.StreamServiceEnabledInterceptor.func1 /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/utils/interceptors/service.go:50 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1 /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1 /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121 github.com/percona/pmm-managed/utils/interceptors.Stream.func1 /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/utils/interceptors/interceptors.go:125 github.com/percona/pmm-managed/utils/interceptors.logRequest /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/utils/interceptors/interceptors.go:78 github.com/percona/pmm-managed/utils/interceptors.Stream /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/utils/interceptors/interceptors.go:122 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1 /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1 /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:58 google.golang.org/grpc.(*Server).processStreamingRPC /home/builder/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1548 google.golang.org/grpc.(*Server).handleStream /home/builder/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:1623 google.golang.org/grpc.(*Server).serveStreams.func1.2 /home/builder/go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:921 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1581 agent_id=pmm-server request=6348dc32-d146-11ec-8397-0242ac110002

I also saw this error many times in the web UI when trying to Upgrade to v2.28.0 (browser console showed: "Failed to load resource: the server responded with a status of 429 (Too Many Requests"):

grpc: received message larger than max (15165483 vs. 5242880)

 

Upgrade log:

ProjectName: pmm-update Version: 2.27.0 PMMVersion: 2.27.0 Timestamp: 2022-04-12 11:11:07 (UTC) FullCommit: 3112681ff16267dad7b883601869a8ea7a6381be Starting "yum --verbose info installed pmm-update" ... Loading "changelog" plugin Loading "fastestmirror" plugin Loading "ovl" plugin Config time: 0.014 rpmdb time: 0.000 ovl: Copying up (0) files from OverlayFS lower layer Yum version: 3.4.3 Installed Packages Name        : pmm-update Arch        : noarch Version     : 2.27.0 Release     : 64.2204121111.3112681.el7 Size        : 2.2 M Repo        : installed From repo   : pmm2-server Committer   : Nikita Beletskii <nikita.beletskii@percona.com> Committime  : Tue Oct 19 12:00:00 2021 Buildtime   : Tue Apr 12 11:11:09 2022 Install time: Mon Apr 25 09:52:53 2022 Installed by: System <unset> Changed by  : System <unset> Summary     : Tool for updating packages and OS configuration for PMM Server URL         : https://github.com/percona/pmm-update License     : AGPLv3 Description : Tool for updating packages and OS configuration for PMM ServerStarting "yum update --assumeyes pmm-update" ... Loaded plugins: changelog, fastestmirror, ovl Loading mirror speeds from cached hostfile  * base: download.cf.centos.org  * epel: d2lzkl7pfhq30w.cloudfront.net  * extras: download.cf.centos.org  * updates: download.cf.centos.org Resolving Dependencies --> Running transaction check ---> Package pmm-update.noarch 0:2.27.0-64.2204121111.3112681.el7 will be updated ---> Package pmm-update.noarch 0:2.28.0-64.2204291517.0e0bfc6.el7 will be an update --> Finished Dependency ResolutionDependencies Resolved================================================================================  Package      Arch     Version                              Repository     Size ================================================================================ Updating:  pmm-update   noarch   2.28.0-64.2204291517.0e0bfc6.el7     pmm2-server   823 kTransaction Summary ================================================================================ Upgrade  1 PackageTotal download size: 823 k Downloading packages: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. Running transaction check Running transaction test Transaction test succeeded Running transaction   Updating   : pmm-update-2.28.0-64.2204291517.0e0bfc6.el7.noarch           1/2    Cleanup    : pmm-update-2.27.0-64.2204121111.3112681.el7.noarch           2/2    Verifying  : pmm-update-2.28.0-64.2204291517.0e0bfc6.el7.noarch           1/2    Verifying  : pmm-update-2.27.0-64.2204121111.3112681.el7.noarch           2/2 Updated:   pmm-update.noarch 0:2.28.0-64.2204291517.0e0bfc6.el7                          Complete! Starting "yum --verbose info installed pmm-update" ... Loading "changelog" plugin Loading "fastestmirror" plugin Loading "ovl" plugin Config time: 0.016 rpmdb time: 0.000 ovl: Copying up (0) files from OverlayFS lower layer Yum version: 3.4.3 Installed Packages Name        : pmm-update Arch        : noarch Version     : 2.28.0 Release     : 64.2204291517.0e0bfc6.el7 Size        : 2.2 M Repo        : installed From repo   : pmm2-server Committer   : Nikita Beletskii <nikita.beletskii@percona.com> Committime  : Tue Oct 19 12:00:00 2021 Buildtime   : Fri Apr 29 15:17:04 2022 Install time: Mon May 16 11:20:35 2022 Installed by: System <unset> Changed by  : System <unset> Summary     : Tool for updating packages and OS configuration for PMM Server URL         : https://github.com/percona/pmm-update License     : AGPLv3 Description : Tool for updating packages and OS configuration for PMM Servertime="2022-05-16T11:20:36Z" level=info msg="pmm-update:\nbefore update = {Version:2.27.0 FullVersion:2.27.0-64.2204121111.3112681.el7 BuildTime:2022-04-12 11:11:09 +0000 UTC Repo:pmm2-server}\n after update = {Version:2.28.0 FullVersion:2.28.0-64.2204291517.0e0bfc6.el7 BuildTime:2022-04-29 15:17:04 +0000 UTC Repo:pmm2-server}" time="2022-05-16T11:20:36Z" level=info msg="Version changed, exiting." ProjectName: pmm-update Version: 2.28.0 PMMVersion: 2.28.0 Timestamp: 2022-04-29 15:17:01 (UTC) FullCommit: 0e0bfc6194bc83292ad4fc6abba869c72b517305 Starting "yum --verbose info installed pmm-update" ... Loading "changelog" plugin Loading "fastestmirror" plugin Loading "ovl" plugin Config time: 0.015 rpmdb time: 0.000 ovl: Copying up (0) files from OverlayFS lower layer Yum version: 3.4.3 Installed Packages Name        : pmm-update Arch        : noarch Version     : 2.28.0 Release     : 64.2204291517.0e0bfc6.el7 Size        : 2.2 M Repo        : installed From repo   : pmm2-server Committer   : Nikita Beletskii <nikita.beletskii@percona.com> Committime  : Tue Oct 19 12:00:00 2021 Buildtime   : Fri Apr 29 15:17:04 2022 Install time: Mon May 16 11:20:35 2022 Installed by: System <unset> Changed by  : System <unset> Summary     : Tool for updating packages and OS configuration for PMM Server URL         : https://github.com/percona/pmm-update License     : AGPLv3 Description : Tool for updating packages and OS configuration for PMM ServerStarting "yum update --assumeyes pmm-update" ... Loaded plugins: changelog, fastestmirror, ovl Loading mirror speeds from cached hostfile  * base: download.cf.centos.org  * epel: d2lzkl7pfhq30w.cloudfront.net  * extras: download.cf.centos.org  * updates: download.cf.centos.org No packages marked for update Starting "yum --verbose info installed pmm-update" ... Loading "changelog" plugin Loading "fastestmirror" plugin Loading "ovl" plugin Config time: 0.022 rpmdb time: 0.000 ovl: Copying up (0) files from OverlayFS lower layer Yum version: 3.4.3 Installed Packages Name        : pmm-update Arch        : noarch Version     : 2.28.0 Release     : 64.2204291517.0e0bfc6.el7 Size        : 2.2 M Repo        : installed From repo   : pmm2-server Committer   : Nikita Beletskii <nikita.beletskii@percona.com> Committime  : Tue Oct 19 12:00:00 2021 Buildtime   : Fri Apr 29 15:17:04 2022 Install time: Mon May 16 11:20:35 2022 Installed by: System <unset> Changed by  : System <unset> Summary     : Tool for updating packages and OS configuration for PMM Server URL         : https://github.com/percona/pmm-update License     : AGPLv3 Description : Tool for updating packages and OS configuration for PMM Servertime="2022-05-16T11:20:39Z" level=info msg="pmm-update:\nbefore update = {Version:2.28.0 FullVersion:2.28.0-64.2204291517.0e0bfc6.el7 BuildTime:2022-04-29 15:17:04 +0000 UTC Repo:pmm2-server}\n after update = {Version:2.28.0 FullVersion:2.28.0-64.2204291517.0e0bfc6.el7 BuildTime:2022-04-29 15:17:04 +0000 UTC Repo:pmm2-server}" time="2022-05-16T11:20:39Z" level=info msg="Version did not change." Starting "ansible-playbook --flush-cache /usr/share/pmm-update/ansible/playbook/tasks/update.yml" ... [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'PLAY [localhost] ***************************************************************TASK [Gathering Facts] ********************************************************* ok: [localhost]TASK [detect /srv/pmm-distribution] ******************************************** ok: [localhost]TASK [detect containers] ******************************************************* ok: [localhost]TASK [force container] ********************************************************* skipping: [localhost]TASK [Remove percona-dashboard without architecture] *************************** ok: [localhost]TASK [Update percona-dashboards package] ***************************************changed: [localhost]TASK [Create grafana config] ***************************************************TASK [Create grafana directory] ************************************************ ok: [localhost] => (item=/srv/grafana) changed: [localhost] => (item=/srv/grafana/plugins)TASK [grafana : Set Grafana folder for plugins on /srv partition for all users] *** ok: [localhost]TASK [Copy new version of grafana.ini] ***************************************** changed: [localhost]TASK [grafana : Create provisioning directory] ********************************* ok: [localhost] => (item=datasources) ok: [localhost] => (item=plugins) ok: [localhost] => (item=dashboards)TASK [Copy grafana provisioning files] ***************************************** ok: [localhost] => (item=datasources) ok: [localhost] => (item=plugins) ok: [localhost] => (item=dashboards)TASK [Upgrade grafana database (Get the latest schema)] ************************ changed: [localhost]TASK [Supervisord start        | Start supervisord service for AMI/OVF] ******** skipping: [localhost]TASK [Check that supervisor socket exists] ************************************* ok: [localhost]TASK [Supervisord start         | Start supervisord for docker] **************** skipping: [localhost]TASK [Run initialization playbook] *********************************************TASK [initialization : Get current version] ************************************ ok: [localhost]TASK [initialization : Get image version] ************************************** ok: [localhost]TASK [initialization : Set current version if VERSION fail doen't exist] ******* skipping: [localhost]TASK [initialization : Setting current and image version] ********************** ok: [localhost]TASK [initialization : Setting current and image version] ********************** ok: [localhost]TASK [initialization : Print current version] ********************************** ok: [localhost] => {     "msg": "Current version: 2.27.0 Image Version: 2.28.0" }TASK [Checking if we need an update or not] ************************************TASK [dashboards_upgrade : Check that old plugin dir exists] ******************* ok: [localhost]TASK [dashboards_upgrade : Find custom plugin in old plugin dir] *************** skipping: [localhost]TASK [dashboards_upgrade : Synchronization plugins (old version before 2.22.0)] *** skipping: [localhost]TASK [dashboards_upgrade : Find custom …

How to test

1. Having a long-running PMM-Instance
2. Upgrade PMM to this FB
3. Now try to Upgrade via UI to another latest version when your step 2 already succeeded

How to document

None

Attachments

2
  • 17 May 2022, 12:17 PM
  • 16 May 2022, 11:29 AM

Smart Checklist

Activity

Show:

Yoni Sade April 4, 2023 at 11:13 AM

After upgrade to v2.36.0 getting the same error:

 

INFO[2023-04-04T11:10:01.205+00:00] RPC /server.Server/Readiness done in 28.364183ms.  request=3e354549-d2d9-11ed-8dd0-0242ac110002 INFO[2023-04-04T11:10:04.438+00:00] Starting RPC /server.Server/Readiness ...     request=402700bd-d2d9-11ed-8dd0-0242ac110002 INFO[2023-04-04T11:10:04.442+00:00] RPC /server.Server/Readiness done in 3.799264ms.  request=402700bd-d2d9-11ed-8dd0-0242ac110002 WARN[2023-04-04T11:10:06.888+00:00] Collect for 49642 buckets took 2.234694129s.  component=qan ERRO[2023-04-04T11:10:06.888+00:00] rpc error: code = ResourceExhausted desc = grpc: received message larger than max (22650814 vs. 20971520) failed to sent CollectRequest to QAN github.com/percona/pmm/managed/services/qan.(*Client).Collect         /home/builder/rpm/BUILD/pmm-8269774a7083e5eb4222c6736c64e321ec7f84c8/src/github.com/percona/pmm/managed/services/qan/client.go:291 github.com/percona/pmm/managed/services/agents.(*Handler).Run.func3         /home/builder/rpm/BUILD/pmm-8269774a7083e5eb4222c6736c64e321ec7f84c8/src/github.com/percona/pmm/managed/services/agents/handler.go:134 runtime/pprof.Do         /usr/local/go/src/runtime/pprof/runtime.go:40 github.com/percona/pmm/managed/services/agents.(*Handler).Run         /home/builder/rpm/BUILD/pmm-8269774a7083e5eb4222c6736c64e321ec7f84c8/src/github.com/percona/pmm/managed/services/agents/handler.go:133 github.com/percona/pmm/managed/services/agents/grpc.(*agentServer).Connect         /home/builder/rpm/BUILD/pmm-8269774a7083e5eb4222c6736c64e321ec7f84c8/src/github.com/percona/pmm/managed/services/agents/grpc/agent_server.go:40 github.com/percona/pmm/api/agentpb._Agent_Connect_Handler         /home/builder/rpm/BUILD/pmm-8269774a7083e5eb4222c6736c64e321ec7f84c8/src/github.com/percona/pmm/api/agentpb/agent_grpc.pb.go:102 github.com/grpc-ecosystem/go-grpc-middleware/validator.StreamServerInterceptor.func1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/validator/validator.go:72 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 github.com/percona/pmm/managed/utils/interceptors.StreamServiceEnabledInterceptor.func1         /home/builder/rpm/BUILD/pmm-8269774a7083e5eb4222c6736c64e321ec7f84c8/src/github.com/percona/pmm/managed/utils/interceptors/service.go:49 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1         /home/builder/go/pkg/mod/github.com/!percona-!lab/go-grpc-prometheus@v0.0.0-20230105215234-10537622c253/server_metrics.go:128 github.com/percona/pmm/managed/utils/interceptors.Stream.func1.1         /home/builder/rpm/BUILD/pmm-8269774a7083e5eb4222c6736c64e321ec7f84c8/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:139 github.com/percona/pmm/managed/utils/interceptors.logRequest         /home/builder/rpm/BUILD/pmm-8269774a7083e5eb4222c6736c64e321ec7f84c8/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:78 github.com/percona/pmm/managed/utils/interceptors.Stream.func1         /home/builder/rpm/BUILD/pmm-8269774a7083e5eb4222c6736c64e321ec7f84c8/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:136 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:58 google.golang.org/grpc.(*Server).processStreamingRPC         /home/builder/go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:1627 google.golang.org/grpc.(*Server).handleStream         /home/builder/go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:1708 google.golang.org/grpc.(*Server).serveStreams.func1.2         /home/builder/go/pkg/mod/google.golang.org/grpc@v1.53.0/server.go:965 runtime.goexit         /usr/local/go/src/runtime/asm_amd64.s:1594  agent_id=pmm-server request=d537c1e4-d2d6-11ed-8dd0-0242ac110002 INFO[2023-04-04T11:10:07.628+00:00] Starting RPC /server.Server/Readiness ...     request=420dba39-d2d9-11ed-8dd0-0242ac110002 INFO[2023-04-04T11:10:07.632+00:00] RPC /server.Server/Readiness done in 3.691483ms.  request=420dba39-d2d9-11ed-8dd0-0242ac110002 INFO[2023-04-04T11:10:10.737+00:00] Starting RPC /server.Server/Readiness ...     request=43e80b0b-d2d9-11ed-8dd0-0242ac110002 INFO[2023-04-04T11:10:10.740+00:00] RPC /server.Server/Readiness done in 3.018471ms.  request=43e80b0b-d2d9-11ed-8dd0-0242ac110002 INFO[2023-04-04T11:10:13.849+00:00] Starting RPC /server.Server/Readiness ...     request=45c301b3-d2d9-11ed-8dd0-0242ac110002

Puneet Kala September 5, 2022 at 2:58 PM

Thanks a lot   for testing and reporting this, I have reopened the ticket for our developers to look at, and we will try to work on this again, we have also seen this again on our internal environments and setups, hence reopening without the need for our support team to re-verify this. 

Yoni Sade September 5, 2022 at 1:54 PM

After upgrade to v2.30.0 still seeing these errors:

ERRO[2022-09-05T13:52:03.518+00:00] rpc error: code = ResourceExhausted desc = grpc: received message larger than max (26986974 vs. 20971520) failed to sent CollectRequest to QAN github.com/percona/pmm/managed/services/qan.(*Client).Collect         /home/builder/rpm/BUILD/pmm-570df2e9e13e2c139c8b3cb6b8dfdf8685a24e45/src/github.com/percona/pmm/managed/services/qan/client.go:273 github.com/percona/pmm/managed/services/agents.(*Handler).Run.func3         /home/builder/rpm/BUILD/pmm-570df2e9e13e2c139c8b3cb6b8dfdf8685a24e45/src/github.com/percona/pmm/managed/services/agents/handler.go:134 runtime/pprof.Do         /usr/local/go/src/runtime/pprof/runtime.go:40 github.com/percona/pmm/managed/services/agents.(*Handler).Run         /home/builder/rpm/BUILD/pmm-570df2e9e13e2c139c8b3cb6b8dfdf8685a24e45/src/github.com/percona/pmm/managed/services/agents/handler.go:133 github.com/percona/pmm/managed/services/agents/grpc.(*agentServer).Connect         /home/builder/rpm/BUILD/pmm-570df2e9e13e2c139c8b3cb6b8dfdf8685a24e45/src/github.com/percona/pmm/managed/services/agents/grpc/agent_server.go:39 github.com/percona/pmm/api/agentpb._Agent_Connect_Handler         /home/builder/rpm/BUILD/pmm-570df2e9e13e2c139c8b3cb6b8dfdf8685a24e45/src/github.com/percona/pmm/api/agentpb/agent_grpc.pb.go:98 github.com/grpc-ecosystem/go-grpc-middleware/validator.StreamServerInterceptor.func1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/validator/validator.go:72 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 github.com/percona/pmm/managed/utils/interceptors.StreamServiceEnabledInterceptor.func1         /home/builder/rpm/BUILD/pmm-570df2e9e13e2c139c8b3cb6b8dfdf8685a24e45/src/github.com/percona/pmm/managed/utils/interceptors/service.go:49 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121 github.com/percona/pmm/managed/utils/interceptors.Stream.func1         /home/builder/rpm/BUILD/pmm-570df2e9e13e2c139c8b3cb6b8dfdf8685a24e45/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:124 github.com/percona/pmm/managed/utils/interceptors.logRequest         /home/builder/rpm/BUILD/pmm-570df2e9e13e2c139c8b3cb6b8dfdf8685a24e45/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:77 github.com/percona/pmm/managed/utils/interceptors.Stream         /home/builder/rpm/BUILD/pmm-570df2e9e13e2c139c8b3cb6b8dfdf8685a24e45/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:121 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:58 google.golang.org/grpc.(*Server).processStreamingRPC         /home/builder/go/pkg/mod/google.golang.org/grpc@v1.48.0/server.go:1565 google.golang.org/grpc.(*Server).handleStream         /home/builder/go/pkg/mod/google.golang.org/grpc@v1.48.0/server.go:1640 google.golang.org/grpc.(*Server).serveStreams.func1.2         /home/builder/go/pkg/mod/google.golang.org/grpc@v1.48.0/server.go:932 runtime.goexit         /usr/local/go/src/runtime/asm_amd64.s:1571  agent_id=pmm-server request=5bc37a1d-2cee-11ed-ba2e-0242ac110002

Yoni Sade August 10, 2022 at 8:25 AM

After upgrade to v2.29.1 still seeing these errors:

 

ERRO[2022-08-10T08:22:04.845+00:00] rpc error: code = ResourceExhausted desc = grpc: received message larger than max (36222461 vs. 20971520) failed to sent CollectRequest to QAN github.com/percona/pmm/managed/services/qan.(*Client).Collect         /home/builder/rpm/BUILD/pmm-2c740d96c9399a5140f5a16b2063024b895753c4/src/github.com/percona/pmm/managed/services/qan/client.go:274 github.com/percona/pmm/managed/services/agents.(*Handler).Run.func3         /home/builder/rpm/BUILD/pmm-2c740d96c9399a5140f5a16b2063024b895753c4/src/github.com/percona/pmm/managed/services/agents/handler.go:135 runtime/pprof.Do         /usr/local/go/src/runtime/pprof/runtime.go:40 github.com/percona/pmm/managed/services/agents.(*Handler).Run         /home/builder/rpm/BUILD/pmm-2c740d96c9399a5140f5a16b2063024b895753c4/src/github.com/percona/pmm/managed/services/agents/handler.go:134 github.com/percona/pmm/managed/services/agents/grpc.(*agentServer).Connect         /home/builder/rpm/BUILD/pmm-2c740d96c9399a5140f5a16b2063024b895753c4/src/github.com/percona/pmm/managed/services/agents/grpc/agent_server.go:40 github.com/percona/pmm/api/agentpb._Agent_Connect_Handler         /home/builder/rpm/BUILD/pmm-2c740d96c9399a5140f5a16b2063024b895753c4/src/github.com/percona/pmm/api/agentpb/agent_grpc.pb.go:98 github.com/grpc-ecosystem/go-grpc-middleware/validator.StreamServerInterceptor.func1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/validator/validator.go:72 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 github.com/percona/pmm/managed/utils/interceptors.StreamServiceEnabledInterceptor.func1         /home/builder/rpm/BUILD/pmm-2c740d96c9399a5140f5a16b2063024b895753c4/src/github.com/percona/pmm/managed/utils/interceptors/service.go:50 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121 github.com/percona/pmm/managed/utils/interceptors.Stream.func1         /home/builder/rpm/BUILD/pmm-2c740d96c9399a5140f5a16b2063024b895753c4/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:125 github.com/percona/pmm/managed/utils/interceptors.logRequest         /home/builder/rpm/BUILD/pmm-2c740d96c9399a5140f5a16b2063024b895753c4/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:78 github.com/percona/pmm/managed/utils/interceptors.Stream         /home/builder/rpm/BUILD/pmm-2c740d96c9399a5140f5a16b2063024b895753c4/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:122 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:49 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1         /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:58 google.golang.org/grpc.(*Server).processStreamingRPC         /home/builder/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:1549 google.golang.org/grpc.(*Server).handleStream         /home/builder/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:1624 google.golang.org/grpc.(*Server).serveStreams.func1.2         /home/builder/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:922 runtime.goexit         /usr/local/go/src/runtime/asm_amd64.s:1571  agent_id=pmm-server request=094b1566-1883-11ed-ad61-0242ac110002

Aaditya Dubey June 27, 2022 at 12:19 PM

Hi  ,

Thank you for the updates.
This issue is getting tracked via https://perconadev.atlassian.net/browse/PMM-9910#icft=PMM-9910 and which is expected to release in 2.29.0, please keep following the same to get updates.   

Done

Details

Assignee

Reporter

Priority

Needs QA

Yes

Needs Doc

Yes

Planned Version/s

Fix versions

Story Points

Affects versions

Environment

pmm-server Docker container on Ubuntu 18.04.3 LTS (2vCPU AMD EPYC 7571 8GB RAM EC2 VM Free mem: 1 GB RAM)

 

Smart Checklist

Created May 16, 2022 at 11:29 AM
Updated March 6, 2024 at 1:12 AM
Resolved November 28, 2022 at 2:46 PM
Configure

Flag notifications