Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

automatic upgrade doesn't upgrade all clusters if operator managing multiple clusters

Description

If operator is managing multiple clusters not all of them get upgraded when there is a new version of PXC. This affects cluster-wide, but also normal deployment where in one namespace operator is managing multiple clusters.
I had two setups.
1. Namespace pxc-test, operator + 3 clusters (named cluster1, cluster2, cluster3)
2. Cluster-wide setup with 3 namespaces (pxc1, pxc2, pxc3), with 1 cluster in each (all named cluster1)

As start deployed operator and all clusters with disabled smart update, but selected "* * * * *" as schedule for new version check.
After deployment patched all 3 clusters like:

kubectl patch pxc cluster1 --type=merge --patch '{"spec": {"upgradeOptions":{ "apply": "recommended" }}}' kubectl patch pxc cluster2 --type=merge --patch '{"spec": {"upgradeOptions":{ "apply": "recommended" }}}' kubectl patch pxc cluster3 --type=merge --patch '{"spec": {"upgradeOptions":{ "apply": "recommended" }}}'

In both cases only one cluster got upgraded.
Initial setup:

NAME READY STATUS RESTARTS AGE cluster1-haproxy-0 2/2 Running 0 20m cluster1-haproxy-1 2/2 Running 0 19m cluster1-haproxy-2 2/2 Running 0 19m cluster1-pxc-0 1/1 Running 0 20m cluster1-pxc-1 1/1 Running 0 19m cluster1-pxc-2 1/1 Running 0 18m cluster2-haproxy-0 2/2 Running 0 20m cluster2-haproxy-1 2/2 Running 0 19m cluster2-haproxy-2 2/2 Running 0 19m cluster2-pxc-0 1/1 Running 0 8m53s cluster2-pxc-1 1/1 Running 0 10m cluster2-pxc-2 1/1 Running 0 11m cluster3-haproxy-0 2/2 Running 0 20m cluster3-haproxy-1 2/2 Running 0 19m cluster3-haproxy-2 2/2 Running 0 18m cluster3-pxc-0 1/1 Running 0 20m cluster3-pxc-1 1/1 Running 0 19m cluster3-pxc-2 1/1 Running 0 18m percona-xtradb-cluster-operator-79d786dcfb-vr4wj 1/1 Running 0 22m

Here's the log output:

{"level":"info","ts":1604078756.67107,"logger":"controller_perconaxtradbcluster","msg":"update PXC version to 8.0.20-11.1 (fetched from db)"} {"level":"info","ts":1604078824.0449321,"logger":"controller_perconaxtradbcluster","msg":"update PXC version to 8.0.20-11.1 (fetched from db)"} {"level":"info","ts":1604078840.949128,"logger":"controller_perconaxtradbcluster","msg":"update PXC version to 8.0.20-11.1 (fetched from db)"} {"level":"info","ts":1604079093.098725,"logger":"controller_perconaxtradbcluster","msg":"add new job: * * * * *"} {"level":"info","ts":1604079097.7654724,"logger":"controller_perconaxtradbcluster","msg":"add new job: * * * * *"} {"level":"info","ts":1604079101.8218434,"logger":"controller_perconaxtradbcluster","msg":"add new job: * * * * *"} {"level":"info","ts":1604079120.413201,"logger":"controller_perconaxtradbcluster","msg":"update PXC version from 8.0.20-11.1 to 8.0.20-11.2"} {"level":"info","ts":1604079122.7721262,"logger":"controller_perconaxtradbcluster","msg":"statefulSet was changed, run smart update"} {"level":"info","ts":1604079122.782112,"logger":"controller_perconaxtradbcluster","msg":"primary pod is cluster2-pxc-0.cluster2-pxc.pxc-test"} {"level":"info","ts":1604079122.7821603,"logger":"controller_perconaxtradbcluster","msg":"apply changes to secondary pod cluster2-pxc-2"} {"level":"info","ts":1604079204.8122394,"logger":"controller_perconaxtradbcluster","msg":"pod cluster2-pxc-2 is running"} {"level":"info","ts":1604079209.8268373,"logger":"controller_perconaxtradbcluster","msg":"apply changes to secondary pod cluster2-pxc-1"} {"level":"info","ts":1604079271.8779588,"logger":"controller_perconaxtradbcluster","msg":"pod cluster2-pxc-1 is running"} {"level":"info","ts":1604079276.889168,"logger":"controller_perconaxtradbcluster","msg":"apply changes to primary pod cluster2-pxc-0"} {"level":"info","ts":1604079340.9120953,"logger":"controller_perconaxtradbcluster","msg":"pod cluster2-pxc-0 is running"} {"level":"info","ts":1604079345.9246433,"logger":"controller_perconaxtradbcluster","msg":"smart update finished"}

And if we check images, only for one cluster is new PXC image:

$ kubectl get pxc cluster1 -o jsonpath='{.spec.pxc.image}' percona/percona-xtradb-cluster:8.0.20-11.1 $ kubectl get pxc cluster2 -o jsonpath='{.spec.pxc.image}' percona/percona-xtradb-cluster:8.0.20-11.2 $ kubectl get pxc cluster3 -o jsonpath='{.spec.pxc.image}' percona/percona-xtradb-cluster:8.0.20-11.1

Environment

None

Smart Checklist

Activity

Done

Details

Assignee

Reporter

Labels

Time tracking

1d 1h logged

Fix versions

Affects versions

Priority

Smart Checklist

Created October 30, 2020 at 6:54 PM
Updated March 5, 2024 at 6:03 PM
Resolved February 2, 2021 at 3:49 PM

Flag notifications