pmm-managed is restarting continuously because of null version of pmm-agent

Description

User impact: User can't upgrade PMM to 2.26.0

STR:

  1. Setup PMM Server 2.26.0

  2. Add pmm-agent via API or via `pmm-admin inventory add agent pmm-agent` command

  3. Restart PMM Server

Actual result: pmm-managed is restarting every few seconds and receiving panic error.

Regarding user's report pmm_agent column has "null" version
Original report:

Since upgrade to PMM 2.26.0, PMM Managed is restarting every few seconds. It is receiving panic runtime error each time. Please see logs below. Error message is “panic: runtime error: invalid memory address or nil pointer dereference”.

INFO[2022-02-24T13:50:17.423+00:00] vmalert configuration not changed, doing nothing. component=supervisord
INFO[2022-02-24T13:50:17.423+00:00] alertmanager configuration not changed, doing nothing. component=supervisord
INFO[2022-02-24T13:50:17.423+00:00] qan-api2 configuration not changed, doing nothing. component=supervisord
INFO[2022-02-24T13:50:17.423+00:00] grafana configuration not changed, doing nothing. component=supervisord
INFO[2022-02-24T13:50:17.423+00:00] dbaas-controller configuration not changed, doing nothing. component=supervisord
INFO[2022-02-24T13:50:17.423+00:00] prometheus configuration not changed, doing nothing. component=supervisord
INFO[2022-02-24T13:50:17.423+00:00] victoriametrics configuration not changed, doing nothing. component=supervisord
INFO[2022-02-24T13:50:17.423+00:00] Checking VictoriaMetrics… component=setup
INFO[2022-02-24T13:50:17.424+00:00] Checking VMAlert… component=setup
INFO[2022-02-24T13:50:17.424+00:00] Checking Alertmanager… component=setup
INFO[2022-02-24T13:50:17.425+00:00] Setup completed. component=setup
INFO[2022-02-24T13:50:17.776+00:00] Starting services… component=main
INFO[2022-02-24T13:50:17.776+00:00] Starting… component=vmalert
INFO[2022-02-24T13:50:17.776+00:00] Starting… component=victoriametrics
INFO[2022-02-24T13:50:17.776+00:00] Starting… component=checks
INFO[2022-02-24T13:50:17.776+00:00] Starting… component=alertmanager
INFO[2022-02-24T13:50:17.776+00:00] Starting server on http://127.0.0.1:7772/ … component=JSON
INFO[2022-02-24T13:50:17.776+00:00] Starting server on http://127.0.0.1:7771/ … component=gRPC
INFO[2022-02-24T13:50:17.776+00:00] Starting server on http://127.0.0.1:7773/debug
Registered handlers:
http://127.0.0.1:7773/debug/metrics
http://127.0.0.1:7773/debug/vars
http://127.0.0.1:7773/debug/requests
http://127.0.0.1:7773/debug/events
http://127.0.0.1:7773/debug/pprof component=debug
INFO[2022-02-24T13:50:17.777+00:00] Using default SaaS host “check.percona.com”.
INFO[2022-02-24T13:50:17.777+00:00] Using SaaS host “check.percona.com”.
INFO[2022-02-24T13:50:17.777+00:00] Environment variable “PERCONA_PLATFORM_API_TIMEOUT” is not set, using “30s” as a default timeout for platform API. component=auth
INFO[2022-02-24T13:50:18.413+00:00] Starting Stream /agent.Agent/Connect … agent_id=pmm-server request=b3a418f8-9578-11ec-8421-1260e5508cc3
INFO[2022-02-24T13:50:18.415+00:00] Connected pmm-agent: &

{ID:pmm-server Version:2.26.0 MetricsPort:0}

. agent_id=pmm-server request=b3a418f8-9578-11ec-8421-1260e5508cc3
INFO[2022-02-24T13:50:18.415+00:00] Starting runStateChangeHandler … agent_id=pmm-server request=b3a418f8-9578-11ec-8421-1260e5508cc3
INFO[2022-02-24T13:50:18.780+00:00] Configuration reloaded. component=vmalert
INFO[2022-02-24T13:50:18.813+00:00] Configuration reloaded. component=alertmanager
INFO[2022-02-24T13:50:18.969+00:00] Done. component=victoriametrics
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11afc0e]

goroutine 948 [running]:
github.com/percona/pmm-managed/services/victoriametrics.AddScrapeConfigs(0xc0006e2180, 0xc0006e2180, 0x14ff92f, 0x8, 0x187b300, 0x0)
/home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/services/victoriametrics/prometheus.go:90 +0x68e
github.com/percona/pmm-managed/services/victoriametrics.(*Service).populateConfig.func1(0xc001030210)
/home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/services/victoriametrics/victoriametrics.go:337 +0x625
gopkg.in/reform%2ev1.(*DB).InTransactionContext(0xc0011f5c01,

{0x18630c8, 0xc00003e038}

, 0x1863138, 0xc0010dcd90)
/home/builder/go/pkg/mod/gopkg.in/reform.v1@v1.5.1/db.go:93 +0xa2
gopkg.in/reform%2ev1.(*DB).InTransaction(…)
/home/builder/go/pkg/mod/gopkg.in/reform.v1@v1.5.1/db.go:74
github.com/percona/pmm-managed/services/victoriametrics.(*Service).populateConfig(0xc000572120, 0xc07e01276e5c34cb)
/home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/services/victoriametrics/victoriametrics.go:322 +0x56
github.com/percona/pmm-managed/services/victoriametrics.(*Service).marshalConfig(0xc0001e8140, 0x0)
/home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/services/victoriametrics/victoriametrics.go:215 +0x25
github.com/percona/pmm-managed/services/victoriametrics.(*Service).updateConfiguration(0xc0001e8140,

{0x1863100, 0xc0005699e0}

)
/home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/services/victoriametrics/victoriametrics.go:157 +0xb5
github.com/percona/pmm-managed/services/victoriametrics.(*Service).Run(0xc0001e8140,

{0x1863138, 0xc0009f85d0}

)
/home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/services/victoriametrics/victoriametrics.go:130 +0x325
main.main.func6()
/home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/main.go:838 +0x65
created by main.main
/home/builder/rpm/BUILD/pmm-managed-6914083707b5478605e7bc816de8fc68b5f511f6/src/github.com/percona/pmm-managed/main.go:836 +0x3fe5

How to test

None

How to document

None

Smart Checklist

Activity

Show:

Nurlan Moldomurov March 14, 2022 at 10:08 AM

Nurlan Moldomurov March 14, 2022 at 10:07 AM

it's a different problem and not related to .

Carlos Salguero March 10, 2022 at 4:15 PM

There is a proposed fix for the case the field is null in PMM-9614

Also this function could be handy for other values:

Postgre JSON functions manual here.

Done

Details

Assignee

Reporter

Priority

Needs QA

Yes

Fix versions

Story Points

Affects versions

Smart Checklist

Created March 4, 2022 at 12:24 PM
Updated March 6, 2024 at 1:28 AM
Resolved April 11, 2022 at 9:48 AM