Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

DDL during SST blocks Donor

Description

With 5.7.22-29.26 and 5.7.24-31.33,  a ddl blocks the donor during a SST.   That wasn't the case with 5.6.  On the donor the receive queue grows very large until the SST completes.  Local writes are also blocked.

 

here's 2 threads on the donor:

{{}}*************************** 11. row ***************************
{{ Id: 14}}
{{ User: root}}
{{ Host: localhost}}
{{ db: test}}
{{ Command: Query}}
{{ Time: 305}}
{{ State: Waiting for backup lock}}
{{ Info: create table test2 (id int)}}
{{ Rows_sent: 0}}
Rows_examined: 0
*************************** 12. row ***************************
{{ Id: 15}}
{{ User: root}}
{{ Host: localhost}}
{{ db: test}}
{{ Command: Query}}
{{ Time: 139}}
{{ State: wsrep: initiating pre-commit for write set (2)}}
{{ Info: insert into testtable (data) values (now())}}
{{ Rows_sent: 0}}
Rows_examined: 0

 

Environment

None

Smart Checklist

Activity

Zsolt Parragi November 26, 2020 at 11:40 AM

Hello

The issue with lock-ddl-per-table is that there are several known issues with it, which can result in inconsistent or failed backups. With an inconsistent backup, it's possible that SST succeeds, but the cluster will detect a conflict later because of inconsistencies between the nodes.

Most of this is explained in .

Some of these issues were fixed as part of and will be part of the next (but not the current) releases.

But not all of them, there are still cases where lock-ddl-per-table won't work correctly, and that's why PXC uses lock-ddl instead.

ethaniel 1 November 26, 2020 at 9:48 AM

I could fix this problem for me with:

Although this is not the recommended way (the devs suggest that I clone the wsrep_sst_xtrabackup-v2 into another one), it didn't cause me any harm in updating the main file.

ethaniel 1 November 26, 2020 at 3:38 AM

Is there a chance that --lock-ddl-per-table could somehow help alleviate this issue? However, I couldn't find a way to pass it to xtrabackup during SST.

ethaniel 1 November 25, 2020 at 8:33 PM

I am upgrading my cluster from 5.6 to 5.7. I have 3 active nodes.

I've upgraded one of them to 5.7 by shutting it down, upgrading the binaries and doing mysql_upgrade. It is also synced up via IST with 2 other nodes which are still on 5.6.

Now, I shut down the remaining 5.6 nodes for good and plan to rebuild them via SST after upgrading binaries to 5.7.

All of my traffic is now going to the remaining 5.7 node, including DDL queries.

When I start SST to create a new 5.7 node, these DDL queries lock the DONOR up completely: https://jira.percona.com/browse/PXC-3489

So, even though this bug is marked as "intended" and "invalid", it's causing me a major headache during upgrading.

It will also cause me a major headache if I will have a catastrophe scenario which will leave me with only 1 node running, not allowing me to rebuild other nodes via SST.

Peter Schwaller August 4, 2020 at 2:28 PM

Closing as Invalid.

Not a Bug

Details

Assignee

Reporter

Time tracking

5h 56m logged

Affects versions

Priority

Smart Checklist

Created January 15, 2019 at 9:47 PM
Updated May 17, 2022 at 3:18 PM
Resolved August 4, 2020 at 2:29 PM