MySQL broken after adding a sidecar to smart-updated cluster

Description

After a cluster is smart-updated from 8.0.23 to 8.0.25, if a sidecar is added, the first re-created pod will have PXC image version 8.0.23. As the data directory has already been updated to 8.0.25, and other nodes are on 8.0.25, MySQL fails to start in the restarted pod.

Steps to reproduce:

  1. Create operator and PXC version 1.10.0, cr has 8.0.23

  2. Wait for smart update to finish, and make sure that describe shows pxc pods using image 8.0.25

  3. Update cr, and uncomment the sidecar definition for pxc

  4. Apply updated cr

  5. PXC pod 2 goes into crashloop

The reason for crashloop is that MySQL fails to start with the following error:

Checking PXC image versions reveals an inconsistency:

To resolve the issue one needs to downscale the operator to 0, modify the statefulset, and wipe the data directory on pxc-2, all of which is not straightforward.

Environment

None

Smart Checklist

Activity

Details

Assignee

Reporter

Affects versions

Priority

Smart Checklist

Created February 16, 2022 at 9:24 AM
Updated September 11, 2024 at 11:08 AM