Assertion `trx->global_seqno() > position_' failed.

General

Escalation

General

Escalation

Description

GDB Info

#0  __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62
#1  0x00000000048cf507 in my_write_core (sig=6) at /mnt/jenkins/workspace/pxc80-build/BUILD_TYPE/debug/Host/min-xenial-x64/mysys/stacktrace.cc:284
#2  0x00000000036a9152 in handle_fatal_signal (sig=6) at /mnt/jenkins/workspace/pxc80-build/BUILD_TYPE/debug/Host/min-xenial-x64/sql/signal_handler.cc:181
#3  <signal handler called>
#4  0x00007f5bebeaa428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#5  0x00007f5bebeac02a in __GI_abort () at abort.c:89
#6  0x00007f5bebea2bd7 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x7f5bd7527b40 "trx->global_seqno() > position_", file=file@entry=0x7f5bd75270e2 "galera/src/certification.cpp", line=line@entry=1199, function=function@entry=0x7f5bd7529fc0 <galera::Certification::append_trx(boost::shared_ptr<galera::TrxHandleSlave> const&)::__PRETTY_FUNCTION__> "galera::Certification::TestResult galera::Certification::append_trx(const TrxHandleSlavePtr&)") at assert.c:92
#7  0x00007f5bebea2c82 in __GI___assert_fail (assertion=0x7f5bd7527b40 "trx->global_seqno() > position_", file=0x7f5bd75270e2 "galera/src/certification.cpp", line=1199, function=0x7f5bd7529fc0 <galera::Certification::append_trx(boost::shared_ptr<galera::TrxHandleSlave> const&)::__PRETTY_FUNCTION__> "galera::Certification::TestResult galera::Certification::append_trx(const TrxHandleSlavePtr&)") at assert.c:101
#8  0x00007f5bd74885d8 in galera::Certification::append_trx (this=0x7f5be27fbb10, trx=...) at galera/src/certification.cpp:1199
#9  0x00007f5bd74cf1a1 in galera::ReplicatorSMM::process_pending_queue (this=0x7f5be27fb000, seqno=720) at galera/src/replicator_smm.cpp:3225
#10 0x00007f5bd74d48c8 in galera::ReplicatorSMM::cert (this=0x7f5be27fb000, trx=0x0, ts=...) at galera/src/replicator_smm.cpp:3343
#11 0x00007f5bd74cf53c in galera::ReplicatorSMM::cert_and_catch (this=0x7f5be27fb000, trx=0x0, ts=...) at galera/src/replicator_smm.cpp:3461
#12 0x00007f5bd74c9ee4 in galera::ReplicatorSMM::process_trx (this=0x7f5be27fb000, recv_ctx=0x7f5bee19cc10, ts_ptr=...) at galera/src/replicator_smm.cpp:2208
#13 0x00007f5bd749e881 in galera::GcsActionSource::process_writeset (this=0x7f5be23fbd80, recv_ctx=0x7f5bee19cc10, act=..., exit_loop=@0x7f5bee19c91a: false) at galera/src/gcs_action_source.cpp:62
#14 0x00007f5bd749ef30 in galera::GcsActionSource::dispatch (this=0x7f5be23fbd80, recv_ctx=0x7f5bee19cc10, act=..., exit_loop=@0x7f5bee19c91a: false) at galera/src/gcs_action_source.cpp:109
#15 0x00007f5bd749f694 in galera::GcsActionSource::process (this=0x7f5be23fbd80, recv_ctx=0x7f5bee19cc10, exit_loop=@0x7f5bee19c91a: false) at galera/src/gcs_action_source.cpp:182
#16 0x00007f5bd74bef1a in galera::ReplicatorSMM::async_recv (this=0x7f5be27fb000, recv_ctx=0x7f5bee19cc10) at galera/src/replicator_smm.cpp:452
#17 0x00007f5bd74f1922 in galera_recv (gh=0x7f5beb459780, recv_ctx=0x7f5bee19cc10) at galera/src/wsrep_provider.cpp:236
#18 0x0000000005386c7e in wsrep::wsrep_provider_v26::run_applier (this=0x7f5bdd7f6e40, applier_ctx=0x7f5bee19cc10) at /mnt/jenkins/workspace/pxc80-build/BUILD_TYPE/debug/Host/min-xenial-x64/wsrep-lib/src/wsrep_provider_v26.cpp:714
#19 0x00000000036ebb84 in wsrep_replication_process (thd=0x7f5bc9c0f000, arg=0x0) at /mnt/jenkins/workspace/pxc80-build/BUILD_TYPE/debug/Host/min-xenial-x64/sql/wsrep_thd.cc:60
#20 0x000000000329fc3b in start_wsrep_THD (arg=0x7f5beb43aa50) at /mnt/jenkins/workspace/pxc80-build/BUILD_TYPE/debug/Host/min-xenial-x64/sql/mysqld.cc:8926
#21 0x00000000049c8c0c in pfs_spawn_thread (arg=0x7f5be23f8820) at /mnt/jenkins/workspace/pxc80-build/BUILD_TYPE/debug/Host/min-xenial-x64/storage/perfschema/pfs.cc:2836
#22 0x00007f5bedbd46ba in start_thread (arg=0x7f5bee19d700) at pthread_create.c:333
#23 0x00007f5bebf7c41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Environment

None

Linked issues

blocks

PXC-2899

Release QA for PXC-8.0.18 RC

Smart Checklist

Activity

Show:

Krunal Bauskar January 17, 2020 at 4:16 AM

commit f41e0cfd2131d64a4ed41258a7c883f80e21f8b5
Author: Krunal Bauskar <krunal.bauskar@percona.com>
Date: Thu Jan 16 16:14:00 2020 +0530

: Assertion `trx->global_seqno() > position_' failed.

Update galera module dependency.

Also fixed missing update of wsrep_cluster_conf_id.

Ramesh Sivaraman January 9, 2020 at 8:47 AM

This issue is reproducible with the latest build 8.0.18

Ramesh Sivaraman November 29, 2019 at 5:30 AM

We can reproduce this issue while upgrading the cluster from 5.7 to 8.0.

Reproducible testcase.

BASEDIR=/ssd/ramesh/pxc57
$BASEDIR/bin/mysqld --no-defaults --initialize-insecure --datadir=/ssd/ramesh/node1
$BASEDIR/bin/mysqld --no-defaults --initialize-insecure --datadir=/ssd/ramesh/node2

$BASEDIR/bin/mysqld --defaults-file=/ssd/ramesh/conf/node1.cnf --datadir=/ssd/ramesh/node1 --basedir=$BASEDIR --wsrep-provider=$BASEDIR/lib/libgalera_smm.so --log-error=/ssd/ramesh/log/node1.err  --wsrep-new-cluster &$BASEDIR/bin/mysqld --defaults-file=/ssd/ramesh/conf/node2.cnf --datadir=/ssd/ramesh/node2 --basedir=$BASEDIR --wsrep-provider=$BASEDIR/lib/libgalera_smm.so --log-error=/ssd/ramesh/log/node2.err &

mysql -uroot -S/ssd/ramesh/node1/mysql.sock -e"create database test"
mysql -uroot -S/ssd/ramesh/node1/mysql.sock -e"create database test_one"
mysql -uroot -S/ssd/ramesh/node1/mysql.sock -e"create database test_two"
 
sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test --mysql-user=root --db-driver=mysql --mysql-socket=/ssd/ramesh/node1/mysql.sock --time=100 --db-ps-mode=disable prepare

sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=1000 --tables=10 --threads=10 --mysql-db=test --mysql-user=root --db-driver=mysql --mysql-socket=/ssd/ramesh/node1/mysql.sock --time=100 --db-ps-mode=disable --time=600 run &

mysqladmin -uroot -S/ssd/ramesh/node2/mysql.sock shutdown

sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_one --mysql-user=root --db-driver=mysql --mysql-socket=/ssd/ramesh/node1/mysql.sock --time=100 --db-ps-mode=disable prepare

sysbench /usr/share/sysbench/oltp_read_write.lua --table-size=100000 --tables=10 --threads=10 --mysql-db=test_two --mysql-user=root --db-driver=mysql --mysql-socket=/ssd/ramesh/node1/mysql.sock --time=100 --db-ps-mode=disable prepare

BASEDIR=/ssd/ramesh/pxc8017

remove "wsrep_sst_auth=root:" from node2.cnf

$BASEDIR/bin/mysqld --defaults-file=/ssd/ramesh/conf/node2.cnf --datadir=/ssd/ramesh/node2 --basedir=$BASEDIR --wsrep-provider=$BASEDIR/lib/libgalera_smm.so --log-error=/ssd/ramesh/log/upgrade_node2.err &

The server will crash while upgrading the server 


CNF

node1.cnf

[mysqld]
wsrep-debug=1
wsrep_node_incoming_address=127.0.0.1
wsrep_node_address=127.0.0.1
innodb_file_per_table
innodb_autoinc_lock_mode=2
innodb_flush_log_at_trx_commit=0
wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth=root:
core-file
log-output=none
wsrep_slave_threads=30
wsrep_cluster_address=gcomm://127.0.0.1:18108,127.0.0.1:18208,127.0.0.1:18308,
log_error_verbosity=3
port=18100
wsrep_provider_options='gmcast.listen_addr=tcp://127.0.0.1:18108;'
socket = /ssd/ramesh/node1/mysql.sock
server_id=11
pxc_encrypt_cluster_traffic = OFF

node2.cnf

[mysqld]
wsrep-debug=1
wsrep_node_incoming_address=127.0.0.1
wsrep_node_address=127.0.0.1
innodb_file_per_table
innodb_autoinc_lock_mode=2
innodb_flush_log_at_trx_commit=0
wsrep_sst_method=xtrabackup-v2
wsrep_sst_auth=root:
core-file
log-output=none
wsrep_slave_threads=30
wsrep_cluster_address=gcomm://127.0.0.1:18108,127.0.0.1:18208,127.0.0.1:18308,
log_error_verbosity=3
port=18200
wsrep_provider_options='gmcast.listen_addr=tcp://127.0.0.1:18208;'
socket = /ssd/ramesh/node2/mysql.sock
server_id=12
pxc_encrypt_cluster_traffic = OFF