Upgrade fails immediately after starting
General
Escalation
General
Escalation
Description
How to test
Steps mentioned above.
Use this FB and watchtower image perconalab/watchtower:PMM-13202-fix-double-unlock
How to document
None
Attachments
1
is blocked by
Activity
Show:

saikumar.vs October 7, 2024 at 8:03 AM
verified, dont see these errors anymore.

Nailya Kutlubaeva July 16, 2024 at 2:02 PM
Started with mentioned FB:
but PMM upgrade failed again with the next error:
Done
Details
Details
Assignee

Reporter

Priority
Labels
Needs QA
Yes
Needs Doc
No
Planned Version/s
Fix versions
Story Points
2
Affects versions
Smart Checklist Progress
0/0 - Done
Smart Checklist
Open Smart Checklist
Smart Checklist

Open Smart Checklist
Created July 2, 2024 at 8:47 AM
Updated December 2, 2024 at 3:05 PM
Resolved October 7, 2024 at 10:43 AM
Steps to reproduce:
Start Watchtower:
docker run -v /var/run/docker.sock:/var/run/docker.sock -e WATCHTOWER_HTTP_API_UPDATE=1 -e WATCHTOWER_HTTP_API_TOKEN=123 --hostname=watchtower --network=pmm-qa docker.io/perconalab/watchtower
Start PMM server:
docker run -d -p 80:8080 -p 443:8443 -p 9000:9000 --volume pmm-data:/srv --name pmm-server --hostname pmm-server --network pmm-qa --restart always -e PMM_DEV_UPDATE_DOCKER_REPO=bupychuk/pmm-server -e PMM_WATCHTOWER_HOST:-
http://watchtower:8080-e PMM_WATCHTOWER_TOKEN:-123 -e PMM_ENABLE_TELEMETRY=0 -e PMM_DEV_PERCONA_PLATFORM_PUBLIC_KEY=RWTg+ZmCCjt7O8eWeAmTLAqW+1ozUbpRSKSwNTmO+exlS5KEIPYWuYdX -e PMM_DEV_PERCONA_PLATFORM_ADDRESS=
https://check-dev.percona.comperconalab/pmm-server:3-dev-latest
Open Home dashboard and start upgrade
Result: Upgrade fails with Unknown error.
pmm-managed.log:
time="2024-07-01T13:54:01.613+00:00" level=error msg="Failed to trigger update" error="received non-OK response: 401" service=updater fatal error: sync: unlock of unlocked mutex goroutine 1407 [running]: sync.fatal({0x1b56d71?, 0xc00137ef38?}) /usr/local/go/src/runtime/panic.go:1007 +0x18 sync.(*Mutex).unlockSlow(0xc000136eb4, 0xffffffff) /usr/local/go/src/sync/mutex.go:229 +0x35 sync.(*Mutex).Unlock(...) /usr/local/go/src/sync/mutex.go:223 github.com/percona/pmm/managed/services/server.(*Updater).StartUpdate(0xc000136ea0, {0x21df158, 0xc0055c8bd0}, {0xc0055be420, 0x1c}) /home/builder/rpm/BUILD/pmm-5d28fccf5eb04909c4e8b35b6e6c86234332a0c7/src/github.com/percona/pmm/managed/services/server/updater.go:163 +0x54f github.com/percona/pmm/managed/services/server.(*Server).StartUpdate(0xc0005c1d40, {0x21df158, 0xc0055c8bd0}, 0xc0055c4cc0) /home/builder/rpm/BUILD/pmm-5d28fccf5eb04909c4e8b35b6e6c86234332a0c7/src/github.com/percona/pmm/managed/services/server/server.go:318 +0x2c9 github.com/percona/pmm/api/serverpb._Server_StartUpdate_Handler.func1({0x21df158?, 0xc0055c8bd0?}, {0x19f8d60?, 0xc0055c4cc0?}) /home/builder/rpm/BUILD/pmm-5d28fccf5eb04909c4e8b35b6e6c86234332a0c7/src/github.com/percona/pmm/api/serverpb/server_grpc.pb.go:310 +0xcb main.runGRPCServer.UnaryServerInterceptor.func7({0x21df158, 0xc0055c8bd0}, {0x19f8d60, 0xc0055c4cc0}, 0x10?, 0xc0054c6828) /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/validator/validator.go:47 +0x65 main.runGRPCServer.ChainUnaryServer.func9.1({0x21df158?, 0xc0055c8bd0?}, {0x19f8d60?, 0xc0055c4cc0?}) /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48 +0x45 main.runGRPCServer.UnaryServiceEnabledInterceptor.func6({0x21df158, 0xc0055c8bd0}, {0x19f8d60, 0xc0055c4cc0}, 0xc0055b81a0, 0xc0055c4d00) /home/builder/rpm/BUILD/pmm-5d28fccf5eb04909c4e8b35b6e6c86234332a0c7/src/github.com/percona/pmm/managed/utils/interceptors/service.go:39 +0x8a main.runGRPCServer.ChainUnaryServer.func9.1({0x21df158?, 0xc0055c8bd0?}, {0x19f8d60?, 0xc0055c4cc0?}) /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:48 +0x45 main.runGRPCServer.(*ServerMetrics).UnaryServerInterceptor.func4({0x21df158, 0xc0055c8bd0}, {0x19f8d60, 0xc0055c4cc0}, 0x4?, 0xc0055c4d40) /home/builder/go/pkg/mod/github.com/!percona-!lab/go-grpc-prometheus@v0.0.0-20230116133345-3487748d4592/server_metrics.go:114 +0x87 main.runGRPCServer.Unary.func5.1() /home/builder/rpm/BUILD/pmm-5d28fccf5eb04909c4e8b35b6e6c86234332a0c7/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:107 +0x58 github.com/percona/pmm/managed/utils/interceptors.logRequest(0xc0055ca070, {0xc0055be0e0, 0x1e}, 0xc0009d5860) /home/builder/rpm/BUILD/pmm-5d28fccf5eb04909c4e8b35b6e6c86234332a0c7/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:78 +0x16b main.runGRPCServer.Unary.func5({0x21df158, 0xc0055c8a80}, {0x19f8d60, 0xc0055c4cc0}, 0xc0055b81a0, 0xc0055c4d40) /home/builder/rpm/BUILD/pmm-5d28fccf5eb04909c4e8b35b6e6c86234332a0c7/src/github.com/percona/pmm/managed/utils/interceptors/interceptors.go:105 +0x345 main.runGRPCServer.ChainUnaryServer.func9({0x21df158, 0xc0055c8a80}, {0x19f8d60, 0xc0055c4cc0}, 0xc0055b81a0, 0x80?) /home/builder/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.4.0/chain.go:53 +0x123 github.com/percona/pmm/api/serverpb._Server_StartUpdate_Handler({0x1aa3800, 0xc0005c1d40}, {0x21df158, 0xc0055c8a80}, 0xc0054c8500, 0xc000c38fc0) /home/builder/rpm/BUILD/pmm-5d28fccf5eb04909c4e8b35b6e6c86234332a0c7/src/github.com/percona/pmm/api/serverpb/server_grpc.pb.go:312 +0x143 google.golang.org/grpc.(*Server).processUnaryRPC(0xc00019d000, {0x21df158, 0xc0055c89f0}, {0x21eeba0, 0xc0001b6f00}, 0xc0055cc240, 0xc000c391a0, 0x30ac4a0, 0x0) /home/builder/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1379 +0xdf8 google.golang.org/grpc.(*Server).handleStream(0xc00019d000, {0x21eeba0, 0xc0001b6f00}, 0xc0055cc240) /home/builder/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1790 +0xe8b google.golang.org/grpc.(*Server).serveStreams.func2.1() /home/builder/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1029 +0x8b created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 361 /home/builder/go/pkg/mod/google.golang.org/grpc@v1.64.0/server.go:1040 +0x125