Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

[WSREP-SST] 2022/01/11 18:06:20 socat[23163] E connect(7, AF=2 10.233.86.195:4444, 16): Connection timed out"}]

Description

First of all, I just wanted to report an issue for your awareness. Not expecting to fix it as I already solved it. May help others in the future too.

If you have any bet on what was happened that would be awesome, otherwise please feel free to close it.

Okay, I have a working PXC operator for a year with 3 nodes and experienced an issue recently.

It could be related with pxc-backup.

Here is what I did:

1) I removed a disk from MinIO. So this service was left running, but obviously not working for some time. It was not possible to push backups to it. This was expected.

2) After some hours XtraDB cluster failed with the errors attached.

I had to do a crash recovery as described in https://www.percona.com/doc/kubernetes-operator-for-pxc/recovery.html

Recovered successfully.

3) After several hours issue repeated

The issue lasted until I removed this block from pxc:

backup: enabled: true image: percona/percona-xtradb-cluster-operator:1.7.0-pxc8.0-backup pitr: enabled: false storageName: STORAGE-NAME-HERE timeBetweenUploads: 60 storages: minio-2: s3: bucket: percona credentialsSecret: db-backup-s3 endpointUrl: http://192.168.240.100:9000 region: us-east-1 type: s3 schedule: - keep: 24 name: minio-backup-2 schedule: '0 8,11,14,17,19 * * *' storageName: minio-2 - keep: 30 name: minio-backup-daily-2 schedule: 0 3 * * * storageName: minio-2 resources: requests: memory: 300M cpu: 80m limits: memory: 400M cpu: 200m

pxc-0

[0] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924097.398457225, {"log"=>"2022-01-11T18:01:37.398319Z 0 [Warning] [MY-000000] [Galera] Could not find peer: f70d35df-72ff-11ec-b7c8-2a60f8408881"}] [1] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924097.398460760, {"log"=>"2022-01-11T18:01:37.398433Z 0 [Warning] [MY-000000] [Galera] 1.0 (db-cluster-pxc-2): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}] [2] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924097.406815231, {"log"=>"2022-01-11T18:01:37.406655Z 0 [Note] [MY-000000] [Galera] Member 1.0 (db-cluster-pxc-2) synced with group."}] [0] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924368.749930875, {"log"=>"2022-01-11T18:06:08.749805Z 0 [Warning] [MY-000000] [Galera] Could not find peer: fde09f94-72ff-11ec-905d-62079652935c"}] [1] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924368.749934058, {"log"=>"2022-01-11T18:06:08.749880Z 0 [Warning] [MY-000000] [Galera] 0.0 (db-cluster-pxc-1): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}] [2] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924368.762522524, {"log"=>"2022-01-11T18:06:08.762425Z 0 [Note] [MY-000000] [Galera] Member 0.0 (db-cluster-pxc-1) synced with group."}] [0] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.263124604, {"log"=>"2022-01-11T18:06:20.262963Z 0 [Note] [MY-000000] [WSREP-SST] 2022/01/11 18:06:20 socat[23163] E connect(7, AF=2 10.233.86.195:4444, 16): Connection timed out"}] [1] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.263793581, {"log"=>"2022-01-11T18:06:20.263599Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR ********************** "}] [2] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.263794935, {"log"=>"2022-01-11T18:06:20.263708Z 0 [ERROR] [MY-000000] [WSREP-SST] Error while sending data to joiner node: exit codes: 0 1"}] [3] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.263795245, {"log"=>"2022-01-11T18:06:20.263722Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 1296"}] [4] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.263795539, {"log"=>"2022-01-11T18:06:20.263755Z 0 [ERROR] [MY-000000] [WSREP-SST] ****************************************************** "}] [5] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.264664175, {"log"=>"2022-01-11T18:06:20.264019Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:32"}] [6] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.284579349, {"log"=>"2022-01-11T18:06:20.276768Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.233.86.195:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.21-12.1' --binlog 'binlog' --gtid '9bde390e-959d-11eb-9e1a-2692c965adf8:750280' : 32 (Broken pipe)"}] [7] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.286411341, {"log"=>"2022-01-11T18:06:20.286301Z 0 [Note] [MY-000000] [Galera] SST sending failed: -32"}] [8] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.286413354, {"log"=>"2022-01-11T18:06:20.286351Z 0 [Note] [MY-000000] [WSREP] Server status change donor -> joined"}] [9] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.286413659, {"log"=>"2022-01-11T18:06:20.286384Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification."}] [10] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.286773476, {"log"=>"2022-01-11T18:06:20.286675Z 0 [ERROR] [MY-000000] [WSREP] Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.233.86.195:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.21-12.1' --binlog 'binlog' --gtid '9bde390e-959d-11eb-9e1a-2692c965adf8:750280' "}] [11] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.298424411, {"log"=>"2022-01-11T18:06:20.298271Z 0 [Warning] [MY-000000] [Galera] Could not find peer: 041c72f6-7300-11ec-bec4-d3dcea7f4868"}] [12] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.298427079, {"log"=>"2022-01-11T18:06:20.298358Z 0 [Warning] [MY-000000] [Galera] 2.0 (db-cluster-pxc-0): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}] [13] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.298427464, {"log"=>"2022-01-11T18:06:20.298377Z 0 [Note] [MY-000000] [Galera] Shifting DONOR/DESYNCED -> JOINED (TO: 750301)"}] [14] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.309189924, {"log"=>"2022-01-11T18:06:20.309043Z 0 [Note] [MY-000000] [Galera] Member 2.0 (db-cluster-pxc-0) synced with group."}] [15] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.309192935, {"log"=>"2022-01-11T18:06:20.309094Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 750301)"}] [16] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.309250287, {"log"=>"2022-01-11T18:06:20.309202Z 10 [Note] [MY-000000] [Galera] Server db-cluster-pxc-0 synced with group"}] [17] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.309264124, {"log"=>"2022-01-11T18:06:20.309252Z 10 [Note] [MY-000000] [WSREP] Server status change joined -> synced"}] [18] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.309280817, {"log"=>"2022-01-11T18:06:20.309267Z 10 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections"}] [19] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.309381695, {"log"=>"2022-01-11T18:06:20.309332Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification."}] $ ~/Documents/git $ k get pods -o wide | grep 10.233.86.195 xb-cron-db-cluster-minio-2-20220111170015-fa5m3-6qzhk 0/1 Error 0 79m 10.233.86.195 node-2.internal <none> <none> $ ~/Documents/git $

pxc-1

[0] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924097.397582985, {"log"=>"2022-01-11T18:01:37.397381Z 0 [Warning] [MY-000000] [Galera] Could not find peer: f70d35df-72ff-11ec-b7c8-2a60f8408881"}] [1] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924097.397586157, {"log"=>"2022-01-11T18:01:37.397532Z 0 [Warning] [MY-000000] [Galera] 1.0 (db-cluster-pxc-2): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}] [2] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924097.407972873, {"log"=>"2022-01-11T18:01:37.407823Z 0 [Note] [MY-000000] [Galera] Member 1.0 (db-cluster-pxc-2) synced with group."}] [0] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.721775962, {"log"=>"2022-01-11T18:06:08.721620Z 0 [Note] [MY-000000] [WSREP-SST] 2022/01/11 18:06:08 socat[11743] E connect(7, AF=2 10.233.86.43:4444, 16): Connection timed out"}] [1] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.723277609, {"log"=>"2022-01-11T18:06:08.722721Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR ********************** "}] [2] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.723279133, {"log"=>"2022-01-11T18:06:08.722789Z 0 [ERROR] [MY-000000] [WSREP-SST] Error while sending data to joiner node: exit codes: 0 1"}] [3] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.723279562, {"log"=>"2022-01-11T18:06:08.722805Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 1296"}] [4] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.723279896, {"log"=>"2022-01-11T18:06:08.722840Z 0 [ERROR] [MY-000000] [WSREP-SST] ****************************************************** "}] [5] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.723280238, {"log"=>"2022-01-11T18:06:08.722979Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:32"}] [6] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.734651033, {"log"=>"2022-01-11T18:06:08.734532Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.233.86.43:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.21-12.1' --binlog 'binlog' --gtid '9bde390e-959d-11eb-9e1a-2692c965adf8:750277' : 32 (Broken pipe)"}] [7] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.736546594, {"log"=>"2022-01-11T18:06:08.736344Z 0 [Note] [MY-000000] [Galera] SST sending failed: -32"}] [8] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.736561514, {"log"=>"2022-01-11T18:06:08.736413Z 0 [Note] [MY-000000] [WSREP] Server status change donor -> joined"}] [9] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.736561903, {"log"=>"2022-01-11T18:06:08.736487Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification."}] [10] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.736587657, {"log"=>"2022-01-11T18:06:08.736547Z 0 [ERROR] [MY-000000] [WSREP] Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.233.86.43:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.21-12.1' --binlog 'binlog' --gtid '9bde390e-959d-11eb-9e1a-2692c965adf8:750277' "}] [11] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.747247838, {"log"=>"2022-01-11T18:06:08.747111Z 0 [Warning] [MY-000000] [Galera] Could not find peer: fde09f94-72ff-11ec-905d-62079652935c"}] [12] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.747250462, {"log"=>"2022-01-11T18:06:08.747161Z 0 [Warning] [MY-000000] [Galera] 0.0 (db-cluster-pxc-1): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}] [13] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.747250859, {"log"=>"2022-01-11T18:06:08.747187Z 0 [Note] [MY-000000] [Galera] Shifting DONOR/DESYNCED -> JOINED (TO: 750300)"}] [14] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.763045407, {"log"=>"2022-01-11T18:06:08.762914Z 0 [Note] [MY-000000] [Galera] Member 0.0 (db-cluster-pxc-1) synced with group."}] [15] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.763048597, {"log"=>"2022-01-11T18:06:08.762977Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 750300)"}] [16] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.763076762, {"log"=>"2022-01-11T18:06:08.763027Z 10 [Note] [MY-000000] [Galera] Server db-cluster-pxc-1 synced with group"}] [17] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.763077274, {"log"=>"2022-01-11T18:06:08.763049Z 10 [Note] [MY-000000] [WSREP] Server status change joined -> synced"}] [18] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.763077591, {"log"=>"2022-01-11T18:06:08.763061Z 10 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections"}] [19] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.763090258, {"log"=>"2022-01-11T18:06:08.763072Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification."}] [0] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924380.295562866, {"log"=>"2022-01-11T18:06:20.295300Z 0 [Warning] [MY-000000] [Galera] Could not find peer: 041c72f6-7300-11ec-bec4-d3dcea7f4868"}] [1] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924380.295566537, {"log"=>"2022-01-11T18:06:20.295397Z 0 [Warning] [MY-000000] [Galera] 2.0 (db-cluster-pxc-0): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}] [2] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924380.308288501, {"log"=>"2022-01-11T18:06:20.308156Z 0 [Note] [MY-000000] [Galera] Member 2.0 (db-cluster-pxc-0) synced with group."}] $ ~/Documents/git $ k get pods -o wide | grep 10.233.86.43 xb-cron-db-cluster-minio-2-20220111170015-fa5m3-xw75t 0/1 Error 0 80m 10.233.86.43 node-2.internal <none> <none> $ ~/Documents/git $

pxc-2

[0] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.352724221, {"log"=>"2022-01-11T18:01:37.352489Z 0 [Note] [MY-000000] [WSREP-SST] 2022/01/11 18:01:37 socat[11943] E connect(7, AF=2 10.233.86.85:4444, 16): Connection timed out"}] [1] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.352727724, {"log"=>"2022-01-11T18:01:37.352558Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR ********************** "}] [2] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.352728057, {"log"=>"2022-01-11T18:01:37.352580Z 0 [ERROR] [MY-000000] [WSREP-SST] Error while sending data to joiner node: exit codes: 0 1"}] [3] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.352728322, {"log"=>"2022-01-11T18:01:37.352590Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 1296"}] [4] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.352728567, {"log"=>"2022-01-11T18:01:37.352598Z 0 [ERROR] [MY-000000] [WSREP-SST] ****************************************************** "}] [5] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.352728814, {"log"=>"2022-01-11T18:01:37.352606Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:32"}] [6] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.370959841, {"log"=>"2022-01-11T18:01:37.370756Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.233.86.85:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.21-12.1' --binlog 'binlog' --gtid '9bde390e-959d-11eb-9e1a-2692c965adf8:750275' : 32 (Broken pipe)"}] [7] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.372589022, {"log"=>"2022-01-11T18:01:37.372466Z 0 [Note] [MY-000000] [Galera] SST sending failed: -32"}] [8] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.372591781, {"log"=>"2022-01-11T18:01:37.372511Z 0 [Note] [MY-000000] [WSREP] Server status change donor -> joined"}] [9] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.372592078, {"log"=>"2022-01-11T18:01:37.372565Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification."}] [10] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.373085260, {"log"=>"2022-01-11T18:01:37.372914Z 0 [ERROR] [MY-000000] [WSREP] Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.233.86.85:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.21-12.1' --binlog 'binlog' --gtid '9bde390e-959d-11eb-9e1a-2692c965adf8:750275' "}] [11] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.390596549, {"log"=>"2022-01-11T18:01:37.390427Z 0 [Warning] [MY-000000] [Galera] Could not find peer: f70d35df-72ff-11ec-b7c8-2a60f8408881"}] [12] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.390599933, {"log"=>"2022-01-11T18:01:37.390481Z 0 [Warning] [MY-000000] [Galera] 1.0 (db-cluster-pxc-2): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}] [13] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.390600264, {"log"=>"2022-01-11T18:01:37.390505Z 0 [Note] [MY-000000] [Galera] Shifting DONOR/DESYNCED -> JOINED (TO: 750281)"}] [14] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.400481070, {"log"=>"2022-01-11T18:01:37.400364Z 0 [Note] [MY-000000] [Galera] Member 1.0 (db-cluster-pxc-2) synced with group."}] [15] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.400484296, {"log"=>"2022-01-11T18:01:37.400410Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 750281)"}] [16] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.400511197, {"log"=>"2022-01-11T18:01:37.400481Z 1 [Note] [MY-000000] [Galera] Server db-cluster-pxc-2 synced with group"}] [17] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.400521258, {"log"=>"2022-01-11T18:01:37.400512Z 1 [Note] [MY-000000] [WSREP] Server status change joined -> synced"}] [18] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.400534875, {"log"=>"2022-01-11T18:01:37.400524Z 1 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections"}] [19] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.400543887, {"log"=>"2022-01-11T18:01:37.400534Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification."}] [0] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924368.748245296, {"log"=>"2022-01-11T18:06:08.748061Z 0 [Warning] [MY-000000] [Galera] Could not find peer: fde09f94-72ff-11ec-905d-62079652935c"}] [1] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924368.748248629, {"log"=>"2022-01-11T18:06:08.748120Z 0 [Warning] [MY-000000] [Galera] 0.0 (db-cluster-pxc-1): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}] [2] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924368.763576717, {"log"=>"2022-01-11T18:06:08.763443Z 0 [Note] [MY-000000] [Galera] Member 0.0 (db-cluster-pxc-1) synced with group."}] [0] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924380.296875310, {"log"=>"2022-01-11T18:06:20.296600Z 0 [Warning] [MY-000000] [Galera] Could not find peer: 041c72f6-7300-11ec-bec4-d3dcea7f4868"}] [1] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924380.296878643, {"log"=>"2022-01-11T18:06:20.296663Z 0 [Warning] [MY-000000] [Galera] 2.0 (db-cluster-pxc-0): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}] [2] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924380.308418261, {"log"=>"2022-01-11T18:06:20.308135Z 0 [Note] [MY-000000] [Galera] Member 2.0 (db-cluster-pxc-0) synced with group."}] $ ~/Documents/git $ k get pods -o wide | grep 10.233.86.85 xb-cron-db-cluster-minio-2-20220111170015-fa5m3-vjrz7 0/1 Error 0 80m 10.233.86.85 node-2.internal <none> <none> $ ~/Documents/git $

Environment

None

Smart Checklist

Activity

Aaditya Dubey March 22, 2024 at 11:27 AM

Hi

We still haven't heard any news from you. So, I assume the issue no longer persists and will close the ticket. If you disagree, reply and create a follow-up new Jira Ticket.

Slava Sarzhan March 14, 2023 at 12:33 PM

Hi do you have any update?

Laimis September 3, 2022 at 3:22 PM

I've got this full pxc crash again. Yeah, I have to update operator and enable backup. This would test exactly same pxc and external conditions, but with latest operator. It would be right test. I have added this task to do list and will test in a while.

Slava Sarzhan August 17, 2022 at 1:35 PM

Did you try to update your operator ? Can you reproduce this issue using the latest version of operator?

Laimis June 15, 2022 at 12:58 PM

Many thanks. Unfortunately, I can afford some time to focus on upgrades (maybe code investigation), to be honest. That's for a while. As I found the solution to remove "backup" block in these cases, it's okay. Will never catch the same issue.

I will keep checking release notes from time to time, anything that is availability and performance related are bonus to do go ahead with upgrade.

Keep up the good work.

If anything needed, please get in touch.

Incomplete

Details

Assignee

Reporter

Labels

Components

Affects versions

Priority

Smart Checklist

Created January 13, 2022 at 10:32 PM
Updated March 22, 2024 at 11:28 AM
Resolved March 22, 2024 at 11:27 AM

Flag notifications