Issues

Select view

Select search mode

 

Assertion `seq <= last_committed()' failure in debug build

Description

The issue is noticed while testing the bug fix, https://perconadev.atlassian.net/browse/PXC-4495 in CUSTOM-183 build branch.

Bug is seen only on debug build. No such issue is seen in release build.

Online node is not affected, only the node where the network is disconnected is affected.

Steps used to reproduce are similar to  steps on https://perconadev.atlassian.net/browse/PXC-3580. Just execute until step 6 and then wait for sometime. after some minutes the issue gets reproduced. for network disconnection used iptable commands instead.

Will update the detailed steps soon.


2024-10-09T19:19:26.690873Z 13 [Note] [MY-000000] [WSREP] wsrep_after_apply 13 2024-10-09T19:19:26.690904Z 16 [Note] [MY-000000] [WSREP] wsrep: committed write set (14774) 2024-10-09T19:19:26.690945Z 16 [Note] [MY-000000] [WSREP] wsrep_after_apply 16 mysqld: /home/parveez.baig/PXC-CUSTOM183/percona-xtradb-cluster-galera/galera/src/replicator_smm.cpp:2314: virtual void galera::ReplicatorSMM::process_commit_cut(wsrep_seqno_t, wsrep_seqno_t): Assertion `seq <= last_committed()' failed. 2024-10-09T19:19:26.691158Z 16 [Note] [MY-000000] [WSREP] Initiating SST cancellation 2024-10-09T19:19:26Z UTC - mysqld got signal 6 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. BuildID[sha1]=59b273a91af38aa6147e1c11877b4bfd66faf029 Server Version: 8.0.34-26-debug Source distribution, wsrep_26.1.4.3 Thread pointer: 0x7f5ae4016540 2024-10-09T19:19:26.691231Z 10 [Note] [MY-000000] [WSREP] Updating WSREPXid: 3d625ff6-8672-11ef-bb17-a25f95e523bf:14776 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... 2024-10-09T19:19:26.691264Z 10 [Note] [MY-000000] [WSREP] Thread with id (10) granted turn to proceed 2024-10-09T19:19:26.691276Z 10 [Note] [MY-000000] [WSREP] Un-Registering thread with id (10) from wsrep group commit queue 2024-10-09T19:19:26.691357Z 14 [Note] [MY-000000] [WSREP] Generating new auto-inc next-value (Current: 181, New: 182) 2024-10-09T19:19:26.691441Z 10 [Note] [MY-000000] [WSREP] wsrep_after_commit: 1, 1, 14776, 0 2024-10-09T19:19:26.691538Z 10 [Note] [MY-000000] [WSREP] wsrep: committed write set (14776) 2024-10-09T19:19:26.691553Z 10 [Note] [MY-000000] [WSREP] wsrep_after_apply 10 2024-10-09T19:19:26.691635Z 14 [Note] [MY-000000] [WSREP] wsrep: applied write set (14778) 2024-10-09T19:19:26.691659Z 14 [Note] [MY-000000] [WSREP] wsrep: committing write set (14778) 2024-10-09T19:19:26.691677Z 14 [Note] [MY-000000] [WSREP] wsrep_before_prepare: 1 2024-10-09T19:19:26.691688Z 14 [Note] [MY-000000] [WSREP] wsrep: preparing to commit write set(14778) stack_bottom = 7f5ac6d4caa8 thread_stack 0x100000 2024-10-09T19:19:26.691977Z 11 [Note] [MY-000000] [WSREP] Updating WSREPXid: 3d625ff6-8672-11ef-bb17-a25f95e523bf:14777 2024-10-09T19:19:26.692012Z 11 [Note] [MY-000000] [WSREP] Thread with id (11) granted turn to proceed 2024-10-09T19:19:26.692023Z 11 [Note] [MY-000000] [WSREP] Un-Registering thread with id (11) from wsrep group commit queue 2024-10-09T19:19:26.692110Z 14 [Note] [MY-000000] [WSREP] wsrep_after_prepare: 1 2024-10-09T19:19:26.692146Z 14 [Note] [MY-000000] [WSREP] wsrep_before_commit: 1, 14778 2024-10-09T19:19:26.692182Z 14 [Note] [MY-000000] [WSREP] wsrep_ordered_commit: 1 2024-10-09T19:19:26.692196Z 14 [Note] [MY-000000] [WSREP] Registering thread with id (14) in wsrep group commit queue 2024-10-09T19:19:26.692204Z 11 [Note] [MY-000000] [WSREP] wsrep_after_commit: 1, 1, 14777, 0 2024-10-09T19:19:26.692255Z 11 [Note] [MY-000000] [WSREP] wsrep: committed write set (14777) 2024-10-09T19:19:26.692268Z 11 [Note] [MY-000000] [WSREP] wsrep_after_apply 11 2024-10-09T19:19:26.692346Z 2 [Note] [MY-000000] [WSREP] wsrep: applying write-set (14779) 2024-10-09T19:19:26.692904Z 2 [Note] [MY-000000] [WSREP] wsrep: updating row for write-set (14779) 2024-10-09T19:19:26.694108Z 2 [Note] [MY-000000] [WSREP] wsrep: updating row for write-set (14779) 2024-10-09T19:19:26.694652Z 2 [Note] [MY-000000] [WSREP] wsrep: deleting row for write-set (14779) 2024-10-09T19:19:26.694672Z 14 [Note] [MY-000000] [WSREP] Updating WSREPXid: 3d625ff6-8672-11ef-bb17-a25f95e523bf:14778 2024-10-09T19:19:26.694693Z 14 [Note] [MY-000000] [WSREP] Thread with id (14) granted turn to proceed 2024-10-09T19:19:26.694703Z 14 [Note] [MY-000000] [WSREP] Un-Registering thread with id (14) from wsrep group commit queue 2024-10-09T19:19:26.694871Z 14 [Note] [MY-000000] [WSREP] wsrep_after_commit: 1, 1, 14778, 0 2024-10-09T19:19:26.694919Z 14 [Note] [MY-000000] [WSREP] wsrep: committed write set (14778) 2024-10-09T19:19:26.694936Z 14 [Note] [MY-000000] [WSREP] wsrep_after_apply 14 2024-10-09T19:19:26.695002Z 12 [Note] [MY-000000] [WSREP] wsrep: applying write-set (14780) 2024-10-09T19:19:26.695007Z 15 [Note] [MY-000000] [WSREP] wsrep: applying write-set (14781) 2024-10-09T19:19:26.695183Z 2 [Note] [MY-000000] [WSREP] wsrep: writing row for write-set (14779) 2024-10-09T19:19:26.695945Z 12 [Note] [MY-000000] [WSREP] wsrep: updating row for write-set (14780) 2024-10-09T19:19:26.696105Z 15 [Note] [MY-000000] [WSREP] wsrep: updating row for write-set (14781) 2024-10-09T19:19:26.696249Z 2 [Note] [MY-000000] [WSREP] Generating new auto-inc next-value (Current: 9643, New: 9644) 2024-10-09T19:19:26.696469Z 2 [Note] [MY-000000] [WSREP] wsrep: applied write set (14779) 2024-10-09T19:19:26.696490Z 2 [Note] [MY-000000] [WSREP] wsrep: committing write set (14779) 2024-10-09T19:19:26.696506Z 2 [Note] [MY-000000] [WSREP] wsrep_before_prepare: 1 2024-10-09T19:19:26.696516Z 2 [Note] [MY-000000] [WSREP] wsrep: preparing to commit write set(14779) 2024-10-09T19:19:26.697160Z 2 [Note] [MY-000000] [WSREP] wsrep_after_prepare: 1 2024-10-09T19:19:26.697207Z 2 [Note] [MY-000000] [WSREP] wsrep_before_commit: 1, 14779 2024-10-09T19:19:26.697240Z 2 [Note] [MY-000000] [WSREP] wsrep_ordered_commit: 1 2024-10-09T19:19:26.697252Z 2 [Note] [MY-000000] [WSREP] Registering thread with id (2) in wsrep group commit queue 2024-10-09T19:19:26.697736Z 12 [Note] [MY-000000] [WSREP] wsrep: updating row for write-set (14780) ... ... 2024-10-09T19:19:26.701332Z 12 [Note] [MY-000000] [WSREP] Registering thread with id (12) in wsrep group commit queue 2024-10-09T19:19:26.701333Z 15 [Note] [MY-000000] [WSREP] wsrep_after_prepare: 1 2024-10-09T19:19:26.701371Z 15 [Note] [MY-000000] [WSREP] wsrep_before_commit: 1, 14781 2024-10-09T19:19:26.701403Z 15 [Note] [MY-000000] [WSREP] wsrep_ordered_commit: 1 2024-10-09T19:19:26.701416Z 15 [Note] [MY-000000] [WSREP] Registering thread with id (15) in wsrep group commit queue 2024-10-09T19:19:26.701912Z 13 [Note] [MY-000000] [WSREP] wsrep: deleting row for write-set (14782) 2024-10-09T19:19:26.702374Z 13 [Note] [MY-000000] [WSREP] wsrep: writing row for write-set (14782) 2024-10-09T19:19:26.702810Z 13 [Note] [MY-000000] [WSREP] Generating new auto-inc next-value (Current: 8609, New: 8610) 2024-10-09T19:19:26.702961Z 13 [Note] [MY-000000] [WSREP] wsrep: applied write set (14782) 2024-10-09T19:19:26.703142Z 13 [Note] [MY-000000] [WSREP] wsrep: committing write set (14782) 2024-10-09T19:19:26.703180Z 13 [Note] [MY-000000] [WSREP] wsrep_before_prepare: 1 2024-10-09T19:19:26.703191Z 13 [Note] [MY-000000] [WSREP] wsrep: preparing to commit write set(14782) 2024-10-09T19:19:26.703266Z 12 [Note] [MY-000000] [WSREP] Updating WSREPXid: 3d625ff6-8672-11ef-bb17-a25f95e523bf:14780 2024-10-09T19:19:26.703288Z 12 [Note] [MY-000000] [WSREP] Thread with id (12) granted turn to proceed 2024-10-09T19:19:26.703298Z 12 [Note] [MY-000000] [WSREP] Un-Registering thread with id (12) from wsrep group commit queue 2024-10-09T19:19:26.703469Z 12 [Note] [MY-000000] [WSREP] wsrep_after_commit: 1, 1, 14780, 0 2024-10-09T19:19:26.703516Z 12 [Note] [MY-000000] [WSREP] wsrep: committed write set (14780) 2024-10-09T19:19:26.703529Z 12 [Note] [MY-000000] [WSREP] wsrep_after_apply 12 2024-10-09T19:19:26.703574Z 13 [Note] [MY-000000] [WSREP] wsrep_after_prepare: 1 2024-10-09T19:19:26.703608Z 13 [Note] [MY-000000] [WSREP] wsrep_before_commit: 1, 14782 2024-10-09T19:19:26.703638Z 13 [Note] [MY-000000] [WSREP] wsrep_ordered_commit: 1 2024-10-09T19:19:26.703651Z 13 [Note] [MY-000000] [WSREP] Registering thread with id (13) in wsrep group commit queue 2024-10-09T19:19:26.705591Z 15 [Note] [MY-000000] [WSREP] Updating WSREPXid: 3d625ff6-8672-11ef-bb17-a25f95e523bf:14781 2024-10-09T19:19:26.705665Z 15 [Note] [MY-000000] [WSREP] Thread with id (15) granted turn to proceed 2024-10-09T19:19:26.705692Z 15 [Note] [MY-000000] [WSREP] Un-Registering thread with id (15) from wsrep group commit queue 2024-10-09T19:19:26.706026Z 15 [Note] [MY-000000] [WSREP] wsrep_after_commit: 1, 1, 14781, 0 2024-10-09T19:19:26.706137Z 15 [Note] [MY-000000] [WSREP] wsrep: committed write set (14781) 2024-10-09T19:19:26.706167Z 15 [Note] [MY-000000] [WSREP] wsrep_after_apply 15 2024-10-09T19:19:26.706334Z 13 [Note] [MY-000000] [WSREP] Updating WSREPXid: 3d625ff6-8672-11ef-bb17-a25f95e523bf:14782 2024-10-09T19:19:26.706356Z 13 [Note] [MY-000000] [WSREP] Thread with id (13) granted turn to proceed 2024-10-09T19:19:26.706365Z 13 [Note] [MY-000000] [WSREP] Un-Registering thread with id (13) from wsrep group commit queue 2024-10-09T19:19:26.706478Z 13 [Note] [MY-000000] [WSREP] wsrep_after_commit: 1, 1, 14782, 0 2024-10-09T19:19:26.706521Z 13 [Note] [MY-000000] [WSREP] wsrep: committed write set (14782) 2024-10-09T19:19:26.706532Z 13 [Note] [MY-000000] [WSREP] wsrep_after_apply 13 /home/parveez.baig/PXC-CUSTOM183/bld/install/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x59) [0x55dc56851504] /home/parveez.baig/PXC-CUSTOM183/bld/install/bin/mysqld(print_fatal_signal(int)+0x3ce) [0x55dc554b0703] /home/parveez.baig/PXC-CUSTOM183/bld/install/bin/mysqld(handle_fatal_signal+0x83) [0x55dc554b092e] /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f5b1be1b520] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c) [0x7f5b1be6f9fc] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16) [0x7f5b1be1b476] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3) [0x7f5b1be017f3] /lib/x86_64-linux-gnu/libc.so.6(+0x2871b) [0x7f5b1be0171b] /lib/x86_64-linux-gnu/libc.so.6(+0x39e96) [0x7f5b1be12e96] /home/parveez.baig/PXC-CUSTOM183/bld/install/lib/libgalera_smm.so(+0x4e36e) [0x7f5b0f31a36e] /home/parveez.baig/PXC-CUSTOM183/bld/install/lib/libgalera_smm.so(+0xa5a90) [0x7f5b0f371a90] /home/parveez.baig/PXC-CUSTOM183/bld/install/lib/libgalera_smm.so(+0xa61a1) [0x7f5b0f3721a1] /home/parveez.baig/PXC-CUSTOM183/bld/install/lib/libgalera_smm.so(+0x4232c) [0x7f5b0f30e32c] /home/parveez.baig/PXC-CUSTOM183/bld/install/lib/libgalera_smm.so(+0x19b64) [0x7f5b0f2e5b64] /home/parveez.baig/PXC-CUSTOM183/bld/install/bin/mysqld(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*)+0x34) [0x55dc578e3fa2] /home/parveez.baig/PXC-CUSTOM183/bld/install/bin/mysqld(+0x3f137ae) [0x55dc555157ae] /home/parveez.baig/PXC-CUSTOM183/bld/install/bin/mysqld(start_wsrep_THD+0x3f0) [0x55dc54fc4307] /home/parveez.baig/PXC-CUSTOM183/bld/install/bin/mysqld(+0x5ac9046) [0x55dc570cb046] /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f5b1be6dac3] /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x7f5b1beff850] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0): Connection ID (thread ID): 16 Status: NOT_KILLED You may download the Percona XtraDB Cluster operations manual by visiting http://www.percona.com/software/percona-xtradb-cluster/ . You may find information in the manual which will help you identify the cause of the crash. Writing a core file

Environment

None

Details

Assignee

Reporter

Needs QA

Yes

Affects versions

Priority

Smart Checklist

Created October 9, 2024 at 7:44 PM
Updated November 12, 2024 at 11:38 AM

Activity

Show:

parveez.baigNovember 12, 2024 at 11:38 AM

Hi

Thanks, will take care from next time.