Manual Crash Recovery interferes with auto recovery even with auto_recovery: false
Description
Environment
AFFECTED CS IDs
Smart Checklist
Activity
Slava Sarzhan May 30, 2021 at 6:01 PM
@Nickolay Ihalainen Starting from 1.7.0 it is necessary to use pxc 'debug' image for recovery procedure. You can use automatic recovery or semi-automatic recovery approach. What does it mean? In case of automatic recovery operator recovers the cluster (choose the node with the latest seqno and force to bootstrap the cluster from this nod). If the user disabled automatic recovery feature (set false in pxc.autoRecovery option in CR) in case of cluster crash user will see the following message from pxc container:
> kubectl logs cluster1-pxc-0 -c pxc
...
#######FULL_PXC_CLUSTER_CRASH:cluster1-pxc-1.cluster1-pxc.default.svc.cluster.local#######
You have the situation of a full PXC cluster crash. In order to restore your PXC cluster, please check the log
from all pods/nodes to find the node with the most recent data (the one with the highest sequence number (seqno).
It is cluster1-pxc-1.cluster1-pxc.default.svc.cluster.local node with sequence number (seqno): 21
Cluster will recover automatically from the crash now.
If you have set spec.pxc.autoRecovery to false, run the following command to recover manually from this node:
kubectl -n default exec cluster1-pxc-1 -c pxc -- sh -c 'kill -s USR1 1'
#######LAST_LINE:cluster1-pxc-1.cluster1-pxc.default.svc.cluster.local:21:#######So, user can perform any manual actions on PXC nodes and then choose from which exactly node to perform the bootstrap using the command from the message
kubectl -n default exec cluster1-pxc-1 -c pxc -- sh -c 'kill -s USR1 1'Particularly in your case you can perform the following actions to restore the cluster:
0) disable autoRecovery (set pxc.autoRecovery: false in CR and apply it or use patch)
1) force pxc-0 to recovery mode by creating /var/lib/mysql/gvwstate.dat file
kubectl exec -it cluster1-pxc-0 -c logs -- touch /var/lib/mysql/gvwstate.dat2) wait till liveness check restarts the pod 0 or restart it manually to save the time using the following command
kubectl delete pods cluster1-pxc-03) now you have the cluster in recovery mode and manual actions can be performed e.g. you can change the passwords for pxc-0 node using the following approach:
kubectl exec -it cluster1-pxc-0 -c pxc -- bash
cat > /var/lib/mysql/init.sql <<EOF
ALTER USER root@'localhost' IDENTIFIED BY 'root_password';
ALTER USER monitor@'%' IDENTIFIED BY 'monitory';
ALTER USER operator@'%' IDENTIFIED BY 'operatoradmin';
EOF
mysqld -u mysql --init-file=/var/lib/mysql/init.sql --wsrep-provider=none --skip-networking --skip-log-bin
from another console:
mysqladmin -uroot -proot_password shutdown4) now the passwords were restored and you can bootstrap the cluster e.g. from node pxc-0 (pxc-0 has the highest seqno)
kubectl -n default exec cluster1-pxc-1 -c pxc -- sh -c 'kill -s USR1 1'The cluster was restored without any debug images.
P.S. Please note that before performing all these commands on the production system you need to re-check everything on your test one.
Manual crash recovery issues:
Create a cluster from cr.yaml wait until ready
Change password on pxc-0:
kubectl exec -it cluster1-pxc-0 -c pxc -- mysql -uroot -proot_password create database test; create table test.t(id int auto_increment primary key); -- disable wsrep and binary log before changing passwords set session wsrep_on=0;set session sql_log_bin=0; ALTER USER clustercheck@localhost IDENTIFIED BY 'secret'; ALTER USER root@'%' IDENTIFIED BY 'secret'; ALTER USER root@localhost IDENTIFIED BY 'secret'; ALTER USER monitor@'%' IDENTIFIED BY 'secret'; ALTER USER operator@'%' IDENTIFIED BY 'secret'; ALTER USER xtrabackup@'%' IDENTIFIED BY 'secret'; #in a second terminal kubectl exec -it cluster1-pxc-0 -c pxc -- mysql -uroot -psecret -e shutdownApply cr.yaml again
Now cluster is broken:
kubectl get pods NAME READY STATUS RESTARTS AGE percona-xtradb-cluster-operator-77bfd8cdc5-c57wv 1/1 Running 0 17m cluster1-haproxy-0 1/2 Running 0 65s cluster1-pxc-0 2/3 Running 0 65s Warning Unhealthy 10s kubelet Readiness probe failed: ERROR 1045 (28000): Access denied for user 'monitor'@'cluster1-pxc-0.cluster1-pxc.default.svc.cluster.local' (using password: YES)I'm trying to use manual crash recovery in order to boostrap from pxc-2
https://www.percona.com/doc/kubernetes-operator-for-pxc/recovery.html
kubectl patch pxc cluster1 --type=merge --patch '{"spec": {"updateStrategy": "OnDelete" }}' kubectl patch pxc cluster1 --type="merge" -p '{"spec":{"pxc":{"image":"percona/percona-xtradb-cluster:8.0.22-13.1-debug"}}}' for i in $(seq 0 $(($(kubectl get pxc cluster1 -o jsonpath='{.spec.pxc.size}')-1))); do kubectl delete pod cluster1-pxc-$i --force --grace-period=0; done for i in $(seq 0 $(($(kubectl get pxc cluster1 -o jsonpath='{.spec.pxc.size}')-1))); do until [[ $(kubectl get pod cluster1-pxc-$i -o jsonpath='{.status.phase}') == 'Running' ]]; do sleep 10; done; kubectl exec cluster1-pxc-$i -- touch /var/lib/mysql/sst_in_progress; doneThe instruction is incorrect:
for i in $(seq $(($(kubectl get pxc cluster1 -o jsonpath='{.spec.pxc.size}')-1))); do pid=$(kubectl exec cluster1-pxc-$i -- ps -C mysqld-ps -o pid=); if [[ -n "$pid" ]]; then kubectl exec cluster1-pxc-$i -c pxc -- kill -9 $pid; fi; done Defaulting container name to logs. Use 'kubectl describe pod/cluster1-pxc-1 -n default' to see all of the containers in this pod. error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "7b8e1dbe6c16a18e81dd974946b4c100fe61ddcf939bb57d524784947645c338": OCI runtime exec failed: exec failed: container_linux.go:370: starting container process caused: exec: "ps": executable file not found in $PATH: unknown Defaulting container name to logs.Fixing with:
for i in $(seq $(($(kubectl get pxc cluster1 -o jsonpath='{.spec.pxc.size}')-1))); do pid=$(kubectl exec cluster1-pxc-$i -c pxc -- ps -C mysqld-ps -o pid=); if [[ -n "$pid" ]]; then kubectl exec cluster1-pxc-$i -c pxc -- kill -9 $pid; fi; doneBut I can't get "Ready" for all pods:
kubectl get pods NAME READY STATUS RESTARTS AGE percona-xtradb-cluster-operator-77bfd8cdc5-c57wv 1/1 Running 0 30m cluster1-pxc-1 3/3 Running 0 8m37s cluster1-pxc-2 3/3 Running 0 6m37s cluster1-haproxy-0 1/2 Running 4 13m cluster1-pxc-0 2/3 Error 3 10mEven with debug image /tmp/recovery-case file is created.
Trying to disable auto recovery:
git diff deploy/cr.yaml|egrep '^[+-]' --- a/deploy/cr.yaml +++ b/deploy/cr.yaml - updateStrategy: SmartUpdate + updateStrategy: OnDelete - image: percona/percona-xtradb-cluster:8.0.22-13.1 - autoRecovery: true + image: percona/percona-xtradb-cluster:8.0.22-13.1-debug + autoRecovery: falseEven after restart with autoRecovery: false pods are continue to be restarted by itself.
removing /tmp/recovery-case in every container and killing containers.
/tmp/recovery-case is created again.
pxc-0 is restarted again and again:
MySQL init process in progress... + for i in {120..0} + echo 'SELECT 1' + mysql --protocol=socket -uoperator -hlocalhost --socket=/tmp/mysql.sock --password= -poperatoradmin + echo 'MySQL init process in progress...' + sleep 1 MySQL init process in progress...pxc-1 and pxc-2 awaiting for automatic recovery: #####################################################FULL_PXC_CLUSTER_CRASH:cluster1-pxc-2.cluster1-pxc.default.svc.cluster.local#####################################################But the automatic recovery is disabled in cr.yaml (manual recovery with -debug image is requested).
I've tried to follow manual recovery instruction blindly, but pxc-2 node is restarted but itself.
bash-4.4$ sed -i 's/safe_to_bootstrap: 0/safe_to_bootstrap: 1/g' /var/lib/mysql/grastate.dat bash-4.4$ sed -i 's/wsrep_cluster_address=.*/wsrep_cluster_address=gcomm:\/\//g' /etc/mysql/node.cnf bash-4.4$ mysqld + touch /tmp/recovery-case + [[ '' == *--skip-networking* ]] + /usr/sbin/mysqld-ps command terminated with exit code 137Before termination I can login to mysqld and it shows 1 node cluster in a primary state.
Describe shows:
Normal Created 30s (x2 over 11m) kubelet Created container pxc Normal Started 30s (x2 over 11m) kubelet Started container pxc