Consider some mitigation for PS-9235
General
Escalation
General
Escalation
Description
Environment
None
AFFECTED CS IDs
CS0044496
is blocked by
Activity
Eleonora Zinchenko November 26, 2024 at 7:01 AM
Eleonora Zinchenko
November 26, 2024 at 7:01 AM
Hi,
For history: the issue was in PS when using Vault v2:
vault secrets enable --version=2 -path=pxc-secret kvThe issue is not reproduced with PXC8.0.37. The pxc POD and cluster starts ok:
% k get pxc cluster2 -oyaml|egrep 'vaultSecretName:|image:'
image: perconalab/percona-xtradb-cluster-operator:main-pxc8.0-backup
image: perconalab/percona-xtradb-cluster-operator:main-haproxy
image: perconalab/percona-xtradb-cluster-operator:main-logcollector
image: perconalab/pmm-client:dev-latest
image: perconalab/percona-xtradb-cluster-operator:main-proxysql
image: perconalab/percona-xtradb-cluster-operator:main-pxc8.0
vaultSecretName: keyring-secret-vault
image: perconalab/percona-xtradb-cluster-operator:main-pxc8.0Started with 1 pod and then increased to 3:
% k get pxc
cluster2 cluster2-haproxy.vault-service ready 1 3 2m52s
% k get pxc
NAME ENDPOINT STATUS PXC PROXYSQL HAPROXY AGE
cluster2 cluster2-haproxy.vault-service ready 3 3 7m46s
% k get pods |grep cluster2
cluster2-haproxy-0 2/2 Running 0 23m
cluster2-haproxy-1 2/2 Running 0 21m
cluster2-haproxy-2 2/2 Running 0 21m
cluster2-pxc-0 3/3 Running 0 23m
cluster2-pxc-1 3/3 Running 0 18m
cluster2-pxc-2 3/3 Running 0 17m
% k logs cluster2-pxc-0 |egrep 'ERROR|vault'
Defaulted container "logs" out of: logs, logrotate, pxc, pxc-init (init)
{"log":"2024-11-26T06:22:17.141812Z 0 [Warning] [MY-011197] [Server] Plugin keyring_vault reported: 'Probing pxc-secret for being a mount point successful - identified kv-v2 secret engine.'\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T06:22:17.168418Z 0 [Warning] [MY-011197] [Server] Plugin keyring_vault reported: 'Vault Server outdated key skipped'\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T06:22:17.168445Z 0 [System] [MY-011197] [Server] Plugin keyring_vault reported: 'Could not read key from Vault.'\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T06:22:17.231834Z 0 [Warning] [MY-011197] [Server] Plugin keyring_vault reported: 'Vault Server outdated key skipped'\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T06:22:17.231861Z 0 [System] [MY-011197] [Server] Plugin keyring_vault reported: 'Could not read key from Vault.'\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T06:22:17.243849Z 0 [Warning] [MY-011197] [Server] Plugin keyring_vault reported: 'Vault Server outdated key skipped'\n","file":"/var/lib/mysql/mysqld-error.log"}
{"log":"2024-11-26T06:22:17.243876Z 0 [System] [MY-011197] [Server] Plugin keyring_vault reported: 'Could not read key from Vault.'\n","file":"/var/lib/mysql/mysqld-error.log"}
PS-9235: Keyring vault fails to work with `binlog_rotate_encryption_master_key_at_startup`Done and the default configuration of operator when TDE is enabled makes the cluster unusable. Please consider some kind of mitigation of it, as without it the cluster fails with:
{"log":"2024-05-17T10:06:51.066436Z 0 [Note] [MY-012922] [InnoDB] Waiting for purge to start\n","file":"/var/lib/mysql/wsrep_recovery_verbose.log"} {"log":"2024-05-17T10:06:51.137676Z 0 [Warning] [MY-011197] [Server] Plugin keyring_vault reported: 'Vault Server response metadata has \"deletion_time\" set'\n","file":"/var/lib/mysql/wsrep_recovery_verbose.log"} {"log":"2024-05-17T10:06:51.137743Z 0 [System] [MY-011197] [Server] Plugin keyring_vault reported: 'Could not read key from Vault.'\n","file":"/var/lib/mysql/wsrep_recovery_verbose.log"} {"log":"2024-05-17T10:06:51.137805Z 0 [ERROR] [MY-013285] [Repl] Failed to store key, please check if keyring is loaded.\n","file":"/var/lib/mysql/wsrep_recovery_verbose.log"} {"log":"2024-05-17T10:06:51.137814Z 0 [ERROR] [MY-013288] [Server] Failed to initialize binlog encryption, please check if keyring is loaded.\n","file":"/var/lib/mysql/wsrep_recovery_verbose.log"}Steps to reproduce:
01) Deploy cluster with size of 1.
02) Deploy cluster with size of 3 and then restart it.