PXC 5.7.26 crashes on row_undo_ins
Description
Environment
AFFECTED CS IDs
is duplicated by
Smart Checklist
Activity
Rafael Chacon Vivas January 7, 2021 at 4:08 PM
Makes sense. Thank you for the clarifications!
Lalit Choudhary January 7, 2021 at 7:14 AM
@Rafael Chacon Vivas
As I tested and mentioned in the previous comment problem issue only visible when we are using log_slow_verbosity is `full` or `innodb, So another option could be to avoid the use of these values with log_slow_verbosity.
I'm able to reproduce the issue with debug build. This is a Percona server bug.
with few tests it evident that issue only reproducible when
log_slow_verbosity is {{`full` }}or {{`innodb }}Ref: PS-5820 fixed in PS 5.7.27
Regarding reproducible test case, yes not consistently reproduciable on release binary that the reason I tested with debug build by compiling it from the source.
Rafael Chacon Vivas January 6, 2021 at 10:03 PM
Hi Lalit.
Thanks for the quick reply. Yup, we are planning to upgrade. We also set the verbosity to minimal to mitigate. From the ticket it seems that this only happens with that setting. Is that assumption correct or do we need to disable slow logs entirely?
Question, how easy was for you to reproduce with the instructions that are provided here? We have been trying to reproduce on dev, but no luck.
Lalit Choudhary January 6, 2021 at 7:07 AM
Hi @Rafael Chacon Vivas
This issue is already fixed in PS/PXC 5.7.27 version. So would suggest upgrade to at least 5.7.27 or better to the latest version.
As a temporary workaround, you can disable slow query logs to avoid a crash.
Rafael Chacon Vivas January 5, 2021 at 10:01 PM
@Lalit -
I think we have run into this issue on one of our systems (we are running 5.7.26-29), however we are trying to reproduce the issue in our development environment and we are struggling to recreate the same conditions.
The instructions from this ticket are not creating the crash on our installation. I wonder what we are missing. Is the intent of interrupting the sysbench to generate rollbacks?
Related to that: are rollbacks required to be happening for this bug to manifest?
Details
Details
Assignee
Reporter
Time tracking
Fix versions
Affects versions
Priority
MediumSmart Checklist
Open Smart Checklist
Smart Checklist
PXC 5.7.26 crashes with the following stack-trace:
#0 0x00007f8234dc39d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x00000000007b5bc5 in handle_fatal_signal (sig=11) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/sql/signal_handler.cc:235 #2 <signal handler called> #3 row_undo_ins (node=0x7f7cb407e6f8, thr=0x7f7cb407e538) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/storage/innobase/row/row0uins.cc:483 #4 0x00000000011136d5 in row_undo (thr=0x7f7cb407e538, node=0x7f7cb407e6f8) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/storage/innobase/row/row0undo.cc:327 #5 row_undo_step (thr=thr@entry=0x7f7cb407e538) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/storage/innobase/row/row0undo.cc:411 #6 0x000000000109a817 in que_thr_step (thr=<optimized out>) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/storage/innobase/que/que0que.cc:1049 #7 que_run_threads_low (thr=<optimized out>) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/storage/innobase/que/que0que.cc:1113 #8 que_run_threads (thr=<optimized out>) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/storage/innobase/que/que0que.cc:1153 #9 0x0000000001168ffb in trx_rollback_to_savepoint_low (trx=trx@entry=0x7f8147c04eb0, savept=savept@entry=0x0) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/storage/innobase/trx/trx0roll.cc:121 #10 0x00000000011699d0 in trx_rollback_for_mysql_low (trx=0x7f8147c04eb0) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/storage/innobase/trx/trx0roll.cc:183 #11 trx_rollback_low (trx=trx@entry=0x7f8147c04eb0) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/storage/innobase/trx/trx0roll.cc:215 #12 0x0000000001169da2 in trx_rollback_for_mysql (trx=0x7f8147c04eb0) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/storage/innobase/trx/trx0roll.cc:292 #13 0x0000000000fd8aa8 in innobase_rollback (hton=<optimized out>, thd=0x7f7cb40318c0, rollback_trx=<optimized out>) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/storage/innobase/handler/ha_innodb.cc:5651 #14 0x0000000000830e9f in ha_rollback_low (thd=thd@entry=0x7f7cb40318c0, all=all@entry=true) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/sql/handler.cc:2071 #15 0x0000000000edc580 in MYSQL_BIN_LOG::rollback (this=0x1e4f1e0 <mysql_bin_log>, thd=0x7f7cb40318c0, all=<optimized out>) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/sql/binlog.cc:2658 #16 0x0000000000831116 in ha_rollback_trans (thd=thd@entry=0x7f7cb40318c0, all=all@entry=true) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/sql/handler.cc:2170 #17 0x0000000000db319e in trans_rollback (thd=thd@entry=0x7f7cb40318c0) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/sql/transaction.cc:389 #18 0x00000000007d1c5a in wsrep_client_rollback (thd=thd@entry=0x7f7cb40318c0) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/sql/wsrep_thd.cc:49 #19 0x0000000000d06bcb in do_command (thd=thd@entry=0x7f7cb40318c0) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/sql/sql_parse.cc:995 #20 0x0000000000dd3508 in handle_connection (arg=arg@entry=0x2525d60) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/sql/conn_handler/connection_handler_per_thread.cc:318 #21 0x0000000000f59ff4 in pfs_spawn_thread (arg=0x250f6f0) at /usr/src/debug/Percona-XtraDB-Cluster-5.7.26-31.37/storage/perfschema/pfs.cc:2190 #22 0x00007f8234dbedd5 in start_thread () from /lib64/libpthread.so.0Could be related/similar to https://jira.percona.com/browse/PS-5727.