commit cbb86224e7b4de0bfdb84fcd3a52967a0ae9fb19
Merge: 62ab7140 c03db52a
Author: Krunal Bauskar <krunal.bauskar@percona.com>
Date: Thu Nov 1 12:21:02 2018 +0530

Merge pull request #148 from percona/3.x-pxc-2220

• PXC#2220: An Xtradb node starts writing to pagestore instead of rin…

commit c03db52adc4aee2c832755c744e4d02a2698fc5e

Author: Krunal Bauskar <krunal.bauskar@percona.com>
Date: Wed Oct 31 08:57:10 2018 +0530

• PXC#2220: An Xtradb node starts writing to pagestore instead of ring bugger

• Issue is triggered if the user try to start parallel instance of
  PXC/mysqld server on a node that already has the instance running.

• Parallel instance eventually fails when it try to bind-address but before
  that galera initialization sequence has altered gcache in
  irrevocable fashion causing the gcache page error.

• Infact, even before gcache is touched, saved state is read.
  Galera existing flow had lock on the grastate.dat (state file).
  Unfortunately, this lock code was buggy or missed a condition
  that re-opening of the file in different mode (through freopen)
  doesn't retain the original lock taken on old file descriptior.

This eventually caused no-lock post freopen and so parallel
instance was allowed to start.

• Fixed the bug by correcting the said condition.

@Shashank Sahni

We have reproduce the issue. Thanks for detailed test-case.

1. Issue would kick in only when you try to start parallel mysqld (with existing one running)

2. So immediately workaround is to avoid doing so.

3. There is check to avoid this from happening but the check is not working as expected. We will fix the issue in upcoming release.

@krunal

1. Thats the point Krunal. The gcache isn't acting as a ring buffer and TXs are spilling to pagestore.

2. I haven't followed this sequence but PFA screenshot charting this value for an affected node. You could see that that value stopped increased at a point and this is exactly when cache spilled to pagestore.

   Open

3. No I didn't.

Have you tried steps I listed in last comment to reproduce it?

There is something that we are missing here.

1. Most of the sysbench workload is lightweight that generate small packets of trx. This means N trx will get appended to galera.cache files and on hitting end of galera cache, cache get used in ring fashion freeing up entries from start of the cache. This sequence continue.

2. You can monitor this sequence as follows

mysql> show status like '%downto%';
----------------------------------+

----------------------------------+

----------------------------------+
1 row in set (0.01 sec)

mysql> show status like '%downto%';
----------------------------------+

----------------------------------+

----------------------------------+
1 row in set (0.00 sec)

mysql> show status like '%downto%';
-----------------------------------+

-----------------------------------+

-----------------------------------+
1 row in set (0.01 sec)

3. Have you freezed purge of gcache that can cause the creation of gcache file even for smaller trx.

Hey Krunal,

I've been able to successfully reproduce the issue in a test environment. This behavior looks like a bug to me. Here are the steps to reproduce it. Please check and share your thoughts.

1. Run a standalone PXC node with following wsrep_provider  options -

2. Generate load. I used sysbench-tpcc benchmark to do so. Let it run for a substantial time to ensure that data being written is more than gcache size.

3. Invoke mysqld. I ran `mysqld --versin`. This logs failure messages 

4. With in few seconds, you should start seeing gcache.page.* files being created and following messages in the log file.