PodDisruptionBudget Problem due to wrong haproxy Statefulset Labels
General
Escalation
General
Escalation
Description
We were using 1.6.0 with extra labels and PodDisruptionBudget as follows:
All was fine until we upgraded to 1.7.0. At this point, we changed the cluster CR to
Along with all other changes needed as per the upgrade documentation.
After the upgrade was over, everything was fine except the statefulset/cluster-haproxy resource which did update the image tag, but kept the old labels in place.
The result is all pods also kept the old 1.6.0-haproxy label and poddisruptionbudget/cluster-haproxy now can't find the pods anymore.
This does not happen with the cluster's StatefulSet which changed it's labels in accordance with the CR specification. It's only a problem for the proxy.
In any case, we can't manually change the PDB resource because the operator changes it back again to what it thinks is the right configuration (the old one).
Environment
None
Smart Checklist
Activity
Aleksandra Abramova August 19, 2021 at 6:46 AM
Sure, you do! We have nice mugs
Davi S Evangelista August 18, 2021 at 2:50 PM
Hello!
A little while back I got some Percona gifts, don't know if I'd qualify for a second round LOL
Thank you anyway for the appreciation : )
Aleksandra Abramova August 18, 2021 at 2:36 PM
Hello! We would like to send you a gift for your active participation!
Please send us an email to community-team@percona.com with info about your current mailing address and phone, and we will ship it to you!
Mykola Marzhan February 11, 2021 at 7:20 AM
thank you for the report - the bug is confirmed. Please escalate the bug via regular channels if your company has a contract with Percona.
Davi S Evangelista February 11, 2021 at 3:39 AM
To solve the PDB problem, we removed the version label in spec.pxc.labels. The old version tag is still in the StatefulSet because it's inside an immutable field. The selector works fine once the operator updates the PDB and we'll just ignore this specific pod label.
As for the operator, the actual error was a network problem with Linkerd and the outbound port 33062/TCP.
We got caught up in unrelated behaviors and thought they were something to report, turns out this issue is not a bug and should be closed.
We were using 1.6.0 with extra labels and PodDisruptionBudget as follows:
All was fine until we upgraded to 1.7.0. At this point, we changed the cluster CR to
Along with all other changes needed as per the upgrade documentation.
After the upgrade was over, everything was fine except the
statefulset/cluster-haproxyresource which did update the image tag, but kept the old labels in place.The result is all pods also kept the old
1.6.0-haproxylabel andpoddisruptionbudget/cluster-haproxynow can't find the pods anymore.This does not happen with the cluster's StatefulSet which changed it's labels in accordance with the CR specification. It's only a problem for the proxy.
In any case, we can't manually change the PDB resource because the operator changes it back again to what it thinks is the right configuration (the old one).