PXC goes non-primary when backup via garbd fails

General

Escalation

General

Escalation

Description

Step to reproduce:

# ./garbd --version
 INFO: 4.3.r752664d

# ./garbd -g 'pxc-cluster' 'gcomm://192.168.1.172' --sst "xtrabackup-v2:192.168.1.119:4444/xtrabackup_sst//1"  --recv-script 'sleep 5 && killall -9 garbd'

Logs from 8.0

2020-09-02T08:12:27.228132Z 0 [Note] [MY-000000] [Galera] (012a0055, 'tcp://0.0.0.0:4567') connection established to 4d5daa0b tcp://192.168.1.119:4567
2020-09-02T08:12:27.228981Z 0 [Note] [MY-000000] [Galera] (012a0055, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
2020-09-02T08:12:27.729658Z 0 [Note] [MY-000000] [Galera] declaring 4d5daa0b at tcp://192.168.1.119:4567 stable
2020-09-02T08:12:27.731017Z 0 [Note] [MY-000000] [Galera] Node 012a0055 state primary
2020-09-02T08:12:27.731185Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(PRIM,012a0055,22)
memb {
	012a0055,0
	4d5daa0b,0
	}
joined {
	}
left {
	}
partitioned {
	}
)
2020-09-02T08:12:27.731206Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
2020-09-02T08:12:27.733485Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
2020-09-02T08:12:27.733565Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 0a4ed3f2-ecf4-11ea-8ec8-03cbddc7c8f4
2020-09-02T08:12:27.742157Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 0a4ed3f2-ecf4-11ea-8ec8-03cbddc7c8f4
2020-09-02T08:12:27.742482Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 0a4ed3f2-ecf4-11ea-8ec8-03cbddc7c8f4 from 0 (pxc-cluster-node-1)
2020-09-02T08:12:27.743704Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 0a4ed3f2-ecf4-11ea-8ec8-03cbddc7c8f4 from 1 (garb)
2020-09-02T08:12:27.743745Z 0 [Note] [MY-000000] [Galera] Quorum results:
	version    = 6,
	component  = PRIMARY,
	conf_id    = 17,
	members    = 1/2 (primary/total),
	act_id     = 19,
	last_appl. = 2,
	protocols  = 2/10/4 (gcs/repl/appl),
	vote policy= 0,
	group UUID = dc2a6da9-ecf2-11ea-8b80-7774be20f500
2020-09-02T08:12:27.743794Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2020-09-02T08:12:27.743867Z 16 [Note] [MY-000000] [Galera] ####### processing CC 20, local, ordered
2020-09-02T08:12:27.743885Z 16 [Note] [MY-000000] [Galera] Drain monitors from 19 upto 19
2020-09-02T08:12:27.743894Z 16 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5, 3)
2020-09-02T08:12:27.743906Z 16 [Note] [MY-000000] [Galera] ####### My UUID: 012a0055-ecf3-11ea-b30e-767813c43e2c
2020-09-02T08:12:27.743918Z 16 [Note] [MY-000000] [Galera] ####### ST not required
2020-09-02T08:12:27.743928Z 16 [Note] [MY-000000] [Galera] Skipping cert index reset
2020-09-02T08:12:27.743941Z 16 [Note] [MY-000000] [Galera] ####### Adjusting cert position: 19 -> 20
2020-09-02T08:12:27.743972Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
2020-09-02T08:12:27.743991Z 16 [Note] [MY-000000] [Galera] ####### Setting monitor position to 20
2020-09-02T08:12:27.744423Z 0 [Note] [MY-000000] [Galera] Member 1.0 (garb) requested state transfer from '*any*'. Selected 0.0 (pxc-cluster-node-1)(SYNCED) as donor.
2020-09-02T08:12:27.744441Z 0 [Note] [MY-000000] [Galera] Shifting SYNCED -> DONOR/DESYNCED (TO: 20)
2020-09-02T08:12:27.747169Z 16 [Note] [MY-000000] [Galera] Recording CC from group: 20
2020-09-02T08:12:27.747200Z 16 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 20
2020-09-02T08:12:27.747208Z 16 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
2020-09-02T08:12:27.747226Z 16 [Note] [MY-000000] [Galera] ================================================
View:
  id: dc2a6da9-ecf2-11ea-8b80-7774be20f500:20
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(2):
	0: 012a0055-ecf3-11ea-b30e-767813c43e2c, pxc-cluster-node-1
	1: 4d5daa0b-ecf3-11ea-9b0c-abf64cce5963, garb
=================================================
2020-09-02T08:12:27.747238Z 16 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2020-09-02T08:12:27.754225Z 16 [Note] [MY-000000] [Galera] Detected STR version: 0, req_len: 51, req: xtrabackup-v2
2020-09-02T08:12:27.754260Z 16 [Note] [MY-000000] [Galera] STR v0: assuming backup request, skipping cert. index preload.
2020-09-02T08:12:27.754293Z 16 [Note] [MY-000000] [WSREP] Server status change synced -> donor
2020-09-02T08:12:27.754305Z 16 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2020-09-02T08:12:27.754393Z 0 [Note] [MY-000000] [WSREP] Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.119:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib/mysql/plugin/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.19-10'   '' --gtid 'dc2a6da9-ecf2-11ea-8b80-7774be20f500:20' )
2020-09-02T08:12:27.777752Z 16 [Note] [MY-000000] [WSREP] DONOR thread signaled with 0
2020-09-02T08:12:27.900941Z 0 [Warning] [MY-000000] [WSREP-SST] wsrep_node_address or wsrep_sst_receive_address not set. Consider setting them if SST fails.
2020-09-02T08:12:30.547099Z 0 [Note] [MY-000000] [Galera] (012a0055, 'tcp://0.0.0.0:4567') turning message relay requesting off
2020-09-02T08:12:33.047888Z 0 [Note] [MY-000000] [Galera] (012a0055, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.1.119:4567 
2020-09-02T08:12:34.048412Z 0 [Note] [MY-000000] [Galera] (012a0055, 'tcp://0.0.0.0:4567') reconnecting to 4d5daa0b (tcp://192.168.1.119:4567), attempt 0
2020-09-02T08:12:37.734401Z 0 [Note] [MY-000000] [Galera] declaring node with index 1 suspected, timeout PT5S (evs.suspect_timeout)
2020-09-02T08:12:37.734464Z 0 [Note] [MY-000000] [Galera] evs::proto(012a0055, OPERATIONAL, view_id(REG,012a0055,22)) suspecting node: 4d5daa0b
2020-09-02T08:12:37.734483Z 0 [Note] [MY-000000] [Galera] evs::proto(012a0055, OPERATIONAL, view_id(REG,012a0055,22)) suspected node without join message, declaring inactive
2020-09-02T08:12:38.235304Z 0 [Note] [MY-000000] [Galera] declaring node with index 1 inactive (evs.inactive_timeout) 
2020-09-02T08:12:38.734834Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,012a0055,22)
memb {
	012a0055,0
	}
joined {
	}
left {
	}
partitioned {
	4d5daa0b,0
	}
)
2020-09-02T08:12:38.734913Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2020-09-02T08:12:38.734928Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
view (view_id(NON_PRIM,012a0055,23)
memb {
	012a0055,0
	}
joined {
	}
left {
	}
partitioned {
	4d5daa0b,0
	}
)
2020-09-02T08:12:38.735003Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2020-09-02T08:12:38.735032Z 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.
2020-09-02T08:12:38.735054Z 0 [Note] [MY-000000] [Galera] Shifting DONOR/DESYNCED -> OPEN (TO: 20)
2020-09-02T08:12:38.735084Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2020-09-02T08:12:38.735124Z 2 [Note] [MY-000000] [Galera] ####### processing CC -1, local, ordered
2020-09-02T08:12:38.735168Z 2 [Note] [MY-000000] [Galera] Drain monitors from 20 upto 20
2020-09-02T08:12:38.735133Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
2020-09-02T08:12:38.735194Z 2 [Note] [MY-000000] [Galera] ####### My UUID: 012a0055-ecf3-11ea-b30e-767813c43e2c
2020-09-02T08:12:38.735216Z 2 [Note] [MY-000000] [Galera] ####### ST not required
2020-09-02T08:12:38.735216Z 0 [Note] [MY-000000] [Galera] Received NON-PRIMARY.
2020-09-02T08:12:38.735249Z 2 [Note] [MY-000000] [Galera] ================================================
View:
  id: dc2a6da9-ecf2-11ea-8b80-7774be20f500:-1
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(1):
	0: 012a0055-ecf3-11ea-b30e-767813c43e2c, pxc-cluster-node-1
=================================================
2020-09-02T08:12:38.735274Z 2 [Note] [MY-000000] [Galera] Non-primary view
2020-09-02T08:12:38.735302Z 2 [Note] [MY-000000] [WSREP] Server status change donor -> connected
2020-09-02T08:12:38.735323Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2020-09-02T08:12:38.735352Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2020-09-02T08:12:38.735398Z 2 [Note] [MY-000000] [Galera] ####### processing CC -1, local, ordered
2020-09-02T08:12:38.735419Z 2 [Note] [MY-000000] [Galera] Drain monitors from 20 upto 20
2020-09-02T08:12:38.735449Z 2 [Note] [MY-000000] [Galera] ####### My UUID: 012a0055-ecf3-11ea-b30e-767813c43e2c
2020-09-02T08:12:38.735468Z 2 [Note] [MY-000000] [Galera] ####### ST not required
2020-09-02T08:12:38.735504Z 2 [Note] [MY-000000] [Galera] ================================================
View:
  id: dc2a6da9-ecf2-11ea-8b80-7774be20f500:-1
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(1):
	0: 012a0055-ecf3-11ea-b30e-767813c43e2c, pxc-cluster-node-1
=================================================
2020-09-02T08:12:38.735520Z 2 [Note] [MY-000000] [Galera] Non-primary view
2020-09-02T08:12:38.735532Z 2 [Note] [MY-000000] [WSREP] Server status change connected -> connected
2020-09-02T08:12:38.735548Z 2 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2020-09-02T08:12:58.136502Z 0 [Note] [MY-000000] [WSREP-SST] 2020/09/02 10:12:58 socat[20042] E connect(7, AF=2 192.168.1.119:4444, 16): Connection refused
2020-09-02T08:12:58.137049Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR ********************** 
2020-09-02T08:12:58.137097Z 0 [ERROR] [MY-000000] [WSREP-SST] Error while sending data to joiner node:  exit codes: 0 1
2020-09-02T08:12:58.137107Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 1206
2020-09-02T08:12:58.137133Z 0 [ERROR] [MY-000000] [WSREP-SST] ****************************************************** 
2020-09-02T08:12:58.137210Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:32
2020-09-02T08:12:58.147818Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.119:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib/mysql/plugin/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.19-10'   '' --gtid 'dc2a6da9-ecf2-11ea-8b80-7774be20f500:20' : 32 (Broken pipe)
2020-09-02T08:12:58.151278Z 0 [Note] [MY-000000] [Galera] SST sending failed: -32
2020-09-02T08:12:58.151309Z 0 [Warning] [MY-000000] [Galera] server: pxc-cluster-node-1 unallowed state transition: connected -> joined

2020-09-02T08:12:58.151317Z 0 [Note] [MY-000000] [WSREP] Server status change connected -> joined
2020-09-02T08:12:58.151333Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification.
2020-09-02T08:12:58.151348Z 0 [ERROR] [MY-000000] [Galera] sst sent called when not SST donor, state CONNECTED
2020-09-02T08:12:58.151358Z 0 [Warning] [MY-000000] [WSREP] Provider sst_sent() returned an error
2020-09-02T08:12:58.151364Z 0 [ERROR] [MY-000000] [WSREP] Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.119:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib/mysql/plugin/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.19-10'   '' --gtid 'dc2a6da9-ecf2-11ea-8b80-7774be20f500:20' 
2020-09-02T08:13:12.063554Z 0 [Note] [MY-000000] [Galera] (012a0055, 'tcp://0.0.0.0:4567') reconnecting to 4d5daa0b (tcp://192.168.1.119:4567), attempt 30

Logs from 5.7:

2020-09-02T08:30:23.300645Z 0 [Note] WSREP: (70e89349, 'tcp://0.0.0.0:4567') connection established to 4d5daa0b tcp://192.168.1.119:4567
2020-09-02T08:30:23.301605Z 0 [Note] WSREP: (70e89349, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
2020-09-02T08:30:23.727046Z 0 [Note] WSREP: declaring 4d5daa0b at tcp://192.168.1.119:4567 stable
2020-09-02T08:30:23.727604Z 0 [Note] WSREP: Node 70e89349 state primary
2020-09-02T08:30:23.736126Z 0 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(PRIM,4d5daa0b,28)
memb {
	4d5daa0b,0
	70e89349,0
	}
joined {
	}
left {
	}
partitioned {
	}
)
2020-09-02T08:30:23.736164Z 0 [Note] WSREP: Save the discovered primary-component to disk
2020-09-02T08:30:23.738816Z 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2020-09-02T08:30:23.738844Z 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2020-09-02T08:30:23.802298Z 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 8baea6e1-ecf6-11ea-b9e8-630213e8e700
2020-09-02T08:30:23.802490Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 8baea6e1-ecf6-11ea-b9e8-630213e8e700 from 1 (pxc-cluster-node-1)
2020-09-02T08:30:23.802877Z 0 [Note] WSREP: STATE EXCHANGE: got state msg: 8baea6e1-ecf6-11ea-b9e8-630213e8e700 from 0 (garb)
2020-09-02T08:30:23.802896Z 0 [Note] WSREP: Quorum results:
	version    = 6,
	component  = PRIMARY,
	conf_id    = 5,
	members    = 1/2 (primary/total),
	act_id     = 0,
	last_appl. = 0,
	protocols  = 0/9/3 (gcs/repl/appl),
	group UUID = 86db0118-ecf5-11ea-b6e4-fadd58830b38
2020-09-02T08:30:23.802905Z 0 [Note] WSREP: Flow-control interval: [100, 100]
2020-09-02T08:30:23.802910Z 0 [Note] WSREP: Trying to continue unpaused monitor
2020-09-02T08:30:23.802971Z 9 [Note] WSREP: REPL Protocols: 9 (4, 2)
2020-09-02T08:30:23.802984Z 9 [Note] WSREP: New cluster view: global state: 86db0118-ecf5-11ea-b6e4-fadd58830b38:0, view# 6: Primary, number of nodes: 2, my index: 1, protocol version 3
2020-09-02T08:30:23.802989Z 9 [Note] WSREP: Setting wsrep_ready to true
2020-09-02T08:30:23.802995Z 9 [Note] WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 1 -> 2) (Increment: 1 -> 2)
2020-09-02T08:30:23.803000Z 9 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-09-02T08:30:23.803007Z 9 [Note] WSREP: Assign initial position for certification: 0, protocol version: 4
2020-09-02T08:30:23.803026Z 0 [Note] WSREP: Service thread queue flushed.
2020-09-02T08:30:23.805439Z 0 [Note] WSREP: Member 0.0 (garb) requested state transfer from '*any*'. Selected 1.0 (pxc-cluster-node-1)(SYNCED) as donor.
2020-09-02T08:30:23.805452Z 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 0)
2020-09-02T08:30:23.805476Z 9 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-09-02T08:30:23.805605Z 0 [Note] WSREP: Initiating SST/IST transfer on DONOR side (wsrep_sst_xtrabackup-v2 --role 'donor' --address '192.168.1.119:4444/xtrabackup_sst//1' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --mysqld-version '5.7.30-33-57'   '' --gtid '86db0118-ecf5-11ea-b6e4-fadd58830b38:0' )
2020-09-02T08:30:23.805933Z 9 [Note] WSREP: DONOR thread signaled with 0
	2020-09-02T08:30:23.945466Z WSREP_SST: [WARNING] wsrep_node_address or wsrep_sst_receive_address not set. Consider setting them if SST fails.
	2020-09-02T08:30:24.141332Z WSREP_SST: [INFO] Streaming with xbstream
2020-09-02T08:30:26.794887Z 0 [Note] WSREP: (70e89349, 'tcp://0.0.0.0:4567') turning message relay requesting off
2020-09-02T08:30:29.295786Z 0 [Note] WSREP: (70e89349, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.1.119:4567 
2020-09-02T08:30:30.296210Z 0 [Note] WSREP: (70e89349, 'tcp://0.0.0.0:4567') reconnecting to 4d5daa0b (tcp://192.168.1.119:4567), attempt 0
2020-09-02T08:30:33.731338Z 0 [Note] WSREP: declaring node with index 0 suspected, timeout PT5S (evs.suspect_timeout)
2020-09-02T08:30:33.731380Z 0 [Note] WSREP: evs::proto(70e89349, OPERATIONAL, view_id(REG,4d5daa0b,28)) suspecting node: 4d5daa0b
2020-09-02T08:30:33.731393Z 0 [Note] WSREP: evs::proto(70e89349, OPERATIONAL, view_id(REG,4d5daa0b,28)) suspected node without join message, declaring inactive
2020-09-02T08:30:34.231676Z 0 [Note] WSREP: declaring node with index 0 inactive (evs.inactive_timeout) 
2020-09-02T08:30:34.731953Z 0 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(NON_PRIM,4d5daa0b,28)
memb {
	70e89349,0
	}
joined {
	}
left {
	}
partitioned {
	4d5daa0b,0
	}
)
2020-09-02T08:30:34.732024Z 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2020-09-02T08:30:34.732042Z 0 [Note] WSREP: Current view of cluster as seen by this node
view (view_id(NON_PRIM,70e89349,29)
memb {
	70e89349,0
	}
joined {
	}
left {
	}
partitioned {
	4d5daa0b,0
	}
)
2020-09-02T08:30:34.732048Z 0 [Note] WSREP: Flow-control interval: [100, 100]
2020-09-02T08:30:34.732057Z 0 [Note] WSREP: Trying to continue unpaused monitor
2020-09-02T08:30:34.732063Z 0 [Note] WSREP: Received NON-PRIMARY.
2020-09-02T08:30:34.732069Z 0 [Note] WSREP: Shifting DONOR/DESYNCED -> OPEN (TO: 0)
2020-09-02T08:30:34.732103Z 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2020-09-02T08:30:34.732121Z 0 [Note] WSREP: Flow-control interval: [100, 100]
2020-09-02T08:30:34.732127Z 0 [Note] WSREP: Trying to continue unpaused monitor
2020-09-02T08:30:34.732133Z 0 [Note] WSREP: Received NON-PRIMARY.
2020-09-02T08:30:34.732142Z 9 [Note] WSREP: New cluster view: global state: 86db0118-ecf5-11ea-b6e4-fadd58830b38:0, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3
2020-09-02T08:30:34.732165Z 9 [Note] WSREP: Setting wsrep_ready to false
2020-09-02T08:30:34.732171Z 9 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2020-09-02T08:30:34.732187Z 9 [Note] WSREP: New cluster view: global state: 86db0118-ecf5-11ea-b6e4-fadd58830b38:0, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3
2020-09-02T08:30:34.732194Z 9 [Note] WSREP: Setting wsrep_ready to false
2020-09-02T08:30:34.732199Z 9 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Environment

None

AFFECTED CS IDs

Linode, ask Mykola Marzhan

Linked issues

relates to

PXC-3538

Garbd crashes after successfull backup

K8SPXC-430

Garbd unsafe for backups

Smart Checklist

Activity

Show:

George Lorch January 28, 2021 at 3:17 PM

https://galeracluster.com/library/documentation/weighted-quorum.html describes this quite well. TL;DR is that garbd backup should always have a pc.weight=0, always.

Iwo Panowicz September 2, 2020 at 5:10 PM
Edited

Yes, as garbd shares the same codebase killing MySQLd during SST would result in similar behaviour. For joiners in particular (please note that this is from fresh nodes thus joiner agrees that it needs to do SST) and forming cluster (or using garbd for backups) I do not believe that's correct behaviour

pc.weight is indeed kind of a workaround for this issue (but only for garbd, not for normal joiners).

Zsolt Parragi September 2, 2020 at 4:43 PM
Edited

Doesn't this also happen with a 2 node cluster (two pxc nodes, no garbd), if the second node that tries to join the cluster fails during SST? I think this is by design.

Setting pc.weight to something greater than one for the actual node should fix this for single-node clusters, that way it should remain primary.

Won't Do

Details
Assignee
Zsolt Parragi
Reporter
Iwo Panowicz
Labels
platform
Time tracking
3d logged
Affects versions
8.0.19-10
5.7.30-31.43
Priority
Medium
Parent
PXC-3532 PXC 8.x Stability Review and Modifications

Smart Checklist

Created September 2, 2020 at 8:31 AM

Updated March 6, 2024 at 9:25 PM

Resolved March 2, 2023 at 9:11 AM

PXC goes non-primary when backup via garbd fails

Description

Environment

AFFECTED CS IDs

Linked issues

relates to

Smart Checklist

Activity

George Lorch January 28, 2021 at 3:17 PM

Iwo Panowicz September 2, 2020 at 5:10 PMEdited

Zsolt Parragi September 2, 2020 at 4:43 PMEdited

DetailsAssigneeZsolt ParragiZsolt ParragiReporterIwo PanowiczIwo PanowiczLabelsplatformTime tracking3d loggedAffects versions8.0.19-105.7.30-31.43PriorityMediumParentPXC-3532 PXC 8.x Stability Review and Modifications

Details

Assignee

Reporter

Labels

Time tracking

Affects versions

Priority

Parent

Smart ChecklistOpen Smart Checklist

Smart Checklist

Iwo Panowicz September 2, 2020 at 5:10 PM
Edited

Zsolt Parragi September 2, 2020 at 4:43 PM
Edited

Details
Assignee
Zsolt Parragi
Reporter
Iwo Panowicz
Labels
platform
Time tracking
3d logged
Affects versions
8.0.19-10
5.7.30-31.43
Priority
Medium
Parent
PXC-3532 PXC 8.x Stability Review and Modifications

Smart Checklist