Skip to:
Create a cluster from cr.yaml, wait until all pods in Stauts=Running.add non-existing option to cr.yaml, e.g.:
spec.pxc configuration: | [mysqld] nonexisting-option=1
Or:tx_isolation=READ-COMMITTED (the variable was removed from mysql 8.0)
In 1.7.0 mysql errors are visible at:
kubectl logs -c logs cluster1-pxc-2 | egrep 'ERROR.*unknown variable'|head -n 1 [10] default.cluster1-pxc-2.mysqld-error.log: [1613012967.426047173, {"log"=>"2021-02-11T03:09:27.423261Z 0 [ERROR] [MY-000067] [Server] unknown variable 'nonexisting-option=1'."}]
There is no error message in pxc container output:
kubectl logs -c pxc cluster1-pxc-2 | egrep 'ERROR.*unknown variable'|head -n 1 kubectl logs --previous -c pxc cluster1-pxc-2 | egrep 'ERROR.*unknown variable'|head -n 1
Trying to fix it:remove nonexisting-option=1, apply cr.yaml, delete pod.Pod still trying to use "nonexisting-option".
My new configuration:configuration: |[mysqld]wait_timeout="wrong value"
But configmap is unchanged:
kubectl get -o yaml configmap cluster1-pxc|grep exist nonexisting-option=1
How to improve:Validate mysql configuration before starting mysql server/before applying cr.yaml to the cluster.
Since mysql 8.0.16:https://dev.mysql.com/doc/refman/8.0/en/server-configuration-validation.html
Unfortunately with current version of PXC it's impossible to validate wsrep options:https://jira.percona.com/browse/PXC-3583
mysqld --wsrep_provider=none --validate-config --log_error_verbosity=2 2021-02-11T03:52:20.858406Z 0 [Warning] [MY-000000] [WSREP] Node is not a cluster node. Disabling pxc_strict_mode 2021-02-11T03:52:20.928095Z 0 [ERROR] [MY-000067] [Server] unknown variable 'nonexisting-option=1'. 2021-02-11T03:52:20.928177Z 0 [ERROR] [MY-010119] [Server] Aborting
Incorrect value for valid variable:wait_timeout="wrong value"
mysqld --wsrep_provider=none --validate-config --log_error_verbosity=2 2021-02-11T04:09:46.148210Z 0 [ERROR] [MY-000058] [Server] Unknown suffix 'w' used for variable 'wait_timeout' (value 'wrong value'). 2021-02-11T04:09:46.148230Z 0 [Warning] [MY-000081] [Server] option 'wait_timeout': unsigned value 0 adjusted to 1. 2021-02-11T04:09:46.148239Z 0 [ERROR] [MY-000077] [Server] /usr/sbin/mysqld: Error while setting value 'wrong value' to 'wait_timeout'. 2021-02-11T04:09:46.148335Z 0 [ERROR] [MY-010119] [Server] Aborting
kubectl logs -c pxc cluster1-pxc-2 is not helpful at all: Just ending at: ++ mysqld --verbose --help --wsrep-provider=none --log-bin-index=/tmp/tmp.bolkL3qriC + DATADIR=/var/lib/mysql kubectl logs -c logs cluster1-pxc-2 | egrep 'mysqld-error.log' [2021/02/11 04:07:49] [ info] [input:tail:tail.0] inotify_fs_add(): inode=28731363 watch_fd=1 name=/var/lib/mysql/mysqld-error.log
The problem severity is high, because even after fixing invalid mysqld configuration SST is forced.
I will turn this ticket from Bug to Improvement.
The goal here would be to have automatic validation of MySQL config.
As we see now it is not possible to do due to https://jira.percona.com/browse/PXC-3583.
We will research other options.
Create a cluster from cr.yaml, wait until all pods in Stauts=Running.
add non-existing option to cr.yaml, e.g.:
spec.pxc configuration: | [mysqld] nonexisting-option=1Or:
tx_isolation=READ-COMMITTED (the variable was removed from mysql 8.0)
In 1.7.0 mysql errors are visible at:
kubectl logs -c logs cluster1-pxc-2 | egrep 'ERROR.*unknown variable'|head -n 1 [10] default.cluster1-pxc-2.mysqld-error.log: [1613012967.426047173, {"log"=>"2021-02-11T03:09:27.423261Z 0 [ERROR] [MY-000067] [Server] unknown variable 'nonexisting-option=1'."}]There is no error message in pxc container output:
kubectl logs -c pxc cluster1-pxc-2 | egrep 'ERROR.*unknown variable'|head -n 1 kubectl logs --previous -c pxc cluster1-pxc-2 | egrep 'ERROR.*unknown variable'|head -n 1Trying to fix it:
remove nonexisting-option=1, apply cr.yaml, delete pod.
Pod still trying to use "nonexisting-option".
My new configuration:
configuration: |
[mysqld]
wait_timeout="wrong value"
But configmap is unchanged:
kubectl get -o yaml configmap cluster1-pxc|grep exist nonexisting-option=1How to improve:
Validate mysql configuration before starting mysql server/before applying cr.yaml to the cluster.
Since mysql 8.0.16:
https://dev.mysql.com/doc/refman/8.0/en/server-configuration-validation.html
Unfortunately with current version of PXC it's impossible to validate wsrep options:
https://jira.percona.com/browse/PXC-3583
mysqld --wsrep_provider=none --validate-config --log_error_verbosity=2 2021-02-11T03:52:20.858406Z 0 [Warning] [MY-000000] [WSREP] Node is not a cluster node. Disabling pxc_strict_mode 2021-02-11T03:52:20.928095Z 0 [ERROR] [MY-000067] [Server] unknown variable 'nonexisting-option=1'. 2021-02-11T03:52:20.928177Z 0 [ERROR] [MY-010119] [Server] AbortingIncorrect value for valid variable:
wait_timeout="wrong value"
mysqld --wsrep_provider=none --validate-config --log_error_verbosity=2 2021-02-11T04:09:46.148210Z 0 [ERROR] [MY-000058] [Server] Unknown suffix 'w' used for variable 'wait_timeout' (value 'wrong value'). 2021-02-11T04:09:46.148230Z 0 [Warning] [MY-000081] [Server] option 'wait_timeout': unsigned value 0 adjusted to 1. 2021-02-11T04:09:46.148239Z 0 [ERROR] [MY-000077] [Server] /usr/sbin/mysqld: Error while setting value 'wrong value' to 'wait_timeout'. 2021-02-11T04:09:46.148335Z 0 [ERROR] [MY-010119] [Server] Abortingkubectl logs -c pxc cluster1-pxc-2 is not helpful at all: Just ending at: ++ mysqld --verbose --help --wsrep-provider=none --log-bin-index=/tmp/tmp.bolkL3qriC + DATADIR=/var/lib/mysql kubectl logs -c logs cluster1-pxc-2 | egrep 'mysqld-error.log' [2021/02/11 04:07:49] [ info] [input:tail:tail.0] inotify_fs_add(): inode=28731363 watch_fd=1 name=/var/lib/mysql/mysqld-error.logThe problem severity is high, because even after fixing invalid mysqld configuration SST is forced.