Looks like there are more corner cases around this area. 

I just found that setting 2 nodes and skipping iptables step (so only sysbench + NBO) is enough for reproducing the problem.

Investigating....

This limitation is not a problem. The problem is that the cluster will be completely blocked, or even the replication plugin will abort. Both behaviors lead to downtime, and the whole cluster has to be restarted to recover the service.

The operation which can't succeed due to limitations should be put into a wait state or rejected with an error, but the cluster should continue with an unrelated workload.

I think it works fine. There is a documented limitation:

So I think it is rather a feature request than a bug?  

This is what happens:

Replicated transaction conflicts with locally executed NBO transaction. This is because NBO transaction releases ApplyOrder monitor after nbo_phase_1. In TOI case apply order is enforced by keeping ApplyOrder monitor for the whole time of TOI, which prevents replicated transaction to be applied and cause BF-BF abort.

How to reproduce (reduced testcase):

create table t1 (a int primary key, b int);

insert into t1 values (0, 0), (1, 1);

1. node_1: 

2. node_2:

We need to pause execution just before NBO applier thread before it starts applying events (and acquiring MDL locks). This is to allow other transaction (point 4) to not wait for any MDL lock and just replicate.

There is no debug sync point there. I've added this just before apply_events() call:

3. node_1:

It waits on execute_command_after_close_tables debug sync point.

4. node_2:

I managed to get a reproducible test case for the NBO-related cluster collapse. Although the behavior differs a bit from what I reported before, I believe the root cause is the same - a BF-BF conflict between NBO and a high-prio thread.

Steps to reproduce:

• Set up PXC 8.0.32 cluster using separate VMs (I used Vagrant and virsh provider, with RockyLinux 9 image)

• Install sysbench on one node and prepare with:

• No particular settings matter, it seems, but I disabled binary logs to avoid other issues related to NBO (unable to rotate):

Follow this sequence of actions:

Second concurrent session on node1:

Now, re-run the sysbench and/or bash loop with CREATE/DROP when they exit with errors.

Repeat that until you see the first BF-BF abort and at this point, you may stop the loops and remove iptables rules with:

At this point, the Galera plugin aborts but mysqld is up and running.

The relevant error log fragments:

node1 log:

node2 log:

node3 log:

Do note that the high-prio thread on node3 is replaying.

Some views into the situation, note that system threads are in killed state:

The only way to recover is to shutdown all nodes and bootstrap the cluster using the best candidate.