Internal server error is returned during PMM3 upgrade
General
Escalation
General
Escalation
Description
Run PMM3 3-dev-latest version and watchtower dev-latest version using pmm3-aws-staging-start job
Open Updates page and run server upgrade
Actual result: Internal server error is returned just in a few seconds
In pmm-managed.log
time="2025-01-13T08:24:15.100+00:00" level=error msg="RPC /server.v1.ServerService/StartUpdate done in 269.971518ms with unexpected error: received non-OK response: 500\ngithub.com/percona/pmm/managed/services/server.(*Updater).sendRequestToWatchtower\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/managed/services/server/updater.go:144\ngithub.com/percona/pmm/managed/services/server.(*Updater).StartUpdate\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/managed/services/server/updater.go:188\ngithub.com/percona/pmm/managed/services/server.(*Server).StartUpdate\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/managed/services/server/server.go:347\ngithub.com/percona/pmm/api/server/v1._ServerService_StartUpdate_Handler.func1\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/api/server/v1/server_grpc.pb.go:353\nmain.runGRPCServer.UnaryServerInterceptor.func7\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/validator/validator.go:47\nmain.runGRPCServer.ChainUnaryServer.func9.1\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\nmain.runGRPCServer.UnaryServiceEnabledInterceptor.func6\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/managed/utils/interceptors/service.go:39\nmain.runGRPCServer.ChainUnaryServer.func9.1\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\nmain.runGRPCServer.(*ServerMetrics).UnaryServerInterceptor.func4\n\t/home/builder/go/pkg/mod/github.com/!percona-!lab/go-grpc-prometheus@v0.0.0-20230116133345-3487748d4592/server_metrics.go:114\nmain.runGRPCServer.Unary.func5.1\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:105\ngithub.com/percona/pmm/managed/utils/interceptors.logRequest\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:77\nmain.runGRPCServer.Unary.func5\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:103\nmain.runGRPCServer.ChainUnaryServer.func9\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:53\ngithub.com/percona/pmm/api/server/v1._ServerService_StartUpdate_Handler\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/api/server/v1/server_grpc.pb.go:355\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/home/builder/go/pkg/mod/google.golang.org/grpc@v1.68.0/server.go:1394\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/home/builder/go/pkg/mod/google.golang.org/grpc@v1.68.0/server.go:1805\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/home/builder/go/pkg/mod/google.golang.org/grpc@v1.68.0/server.go:1029\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700" request=c63ad28d-d187-11ef-be54-0242ac120003
How to test
Create PMM instance with slightly older image (feature build or dev-latest).
Attempt to update, update should be successfully.
If there’s another attempt to update after the first one is successful, the error message should be more helpful (e.g No new image available).
Run PMM3 3-dev-latest version and watchtower dev-latest version using pmm3-aws-staging-start job
Open Updates page and run server upgrade
Actual result: Internal server error is returned just in a few seconds
In pmm-managed.log
time="2025-01-13T08:24:15.100+00:00" level=error msg="RPC /server.v1.ServerService/StartUpdate done in 269.971518ms with unexpected error: received non-OK response: 500\ngithub.com/percona/pmm/managed/services/server.(*Updater).sendRequestToWatchtower\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/managed/services/server/updater.go:144\ngithub.com/percona/pmm/managed/services/server.(*Updater).StartUpdate\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/managed/services/server/updater.go:188\ngithub.com/percona/pmm/managed/services/server.(*Server).StartUpdate\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/managed/services/server/server.go:347\ngithub.com/percona/pmm/api/server/v1._ServerService_StartUpdate_Handler.func1\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/api/server/v1/server_grpc.pb.go:353\nmain.runGRPCServer.UnaryServerInterceptor.func7\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/validator/validator.go:47\nmain.runGRPCServer.ChainUnaryServer.func9.1\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\nmain.runGRPCServer.UnaryServiceEnabledInterceptor.func6\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/managed/utils/interceptors/service.go:39\nmain.runGRPCServer.ChainUnaryServer.func9.1\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48\nmain.runGRPCServer.(*ServerMetrics).UnaryServerInterceptor.func4\n\t/home/builder/go/pkg/mod/github.com/!percona-!lab/go-grpc-prometheus@v0.0.0-20230116133345-3487748d4592/server_metrics.go:114\nmain.runGRPCServer.Unary.func5.1\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:105\ngithub.com/percona/pmm/managed/utils/interceptors.logRequest\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:77\nmain.runGRPCServer.Unary.func5\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:103\nmain.runGRPCServer.ChainUnaryServer.func9\n\t/home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:53\ngithub.com/percona/pmm/api/server/v1._ServerService_StartUpdate_Handler\n\t/home/builder/rpm/BUILD/pmm-07df3de38d2039427775350b90f6d2e4f4fc6f1d/src/github.com/percona/pmm/api/server/v1/server_grpc.pb.go:355\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/home/builder/go/pkg/mod/google.golang.org/grpc@v1.68.0/server.go:1394\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/home/builder/go/pkg/mod/google.golang.org/grpc@v1.68.0/server.go:1805\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/home/builder/go/pkg/mod/google.golang.org/grpc@v1.68.0/server.go:1029\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700" request=c63ad28d-d187-11ef-be54-0242ac120003