crash on snapshot size check in RocksDB
Description
Environment
is blocked by
relates to
Smart Checklist
Activity
George Lorch November 20, 2018 at 9:08 PM
Due to https://perconadev.atlassian.net/browse/PS-5049#icft=PS-5049 and the reversion, this is still an issue until we perform another upstream merge.
George Lorch September 13, 2018 at 5:23 PMEdited
PS 8.0.12
rocksdb.autoinc_debug
rocksdb.drop_index_inplace
rocksdb.nonflushing_analyze_debug
rocksdb.nonflushing_analyze_parts_debug
George Lorch August 1, 2018 at 7:13 PM
It seems this commit in rocksdb is the fix for this https://github.com/facebook/rocksdb/commit/bb2a2ec7313e9af648fc9ac613289e18ed019eb0
George Lorch August 1, 2018 at 5:22 PMEdited
So the first difference is that in 8.0, TC_BINLOG is now being used by default since binlog is enabled by default in 8.0. In 5.7 it is not and thus TC_MMAP is being used. What this means is that the commit process in 8.0 is using the Group Commit 2PC process, i.e. calling prepare, then commit whereas in 5.7 there is no prepare phase. This shows a difference in the path taken within ./storage/rocksdb/rocksdb/utilities/transactions/pessimistic_trancaction.cc within PessimisticTransaction::Rollback()
Enabling binlog in 5.7 reproduces the issue with the same stack trace.
This does not happen on upstream Facebook MySQL 5.6. It appears that the binlog group commit algorithm is not coming into play and is never calling 'prepare'.
This DOES happen on upstream fb-mysql at the fb-prod201801 tag that we are merged up to and utilizing the same rocksdb submodule commit pointer that we have as suggested by Facebook. There have been changes to the 2pc transaction model since that commit. From here I will do two different things, 1) try rebuilding PS with an updated rocksdb submodule that passes through these changes; 2) report this to the upstream team for confirmation/clarification of the issue.
The following crash takes place on RocksDB debug build 8.0.11:
Thread 44 "mysqld-debug" received signal SIGABRT, Aborted. [Switching to Thread 0x7fffc89b9700 (LWP 18428)] 0x00007ffff5151428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 0x00007ffff5151428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 #1 0x00007ffff515302a in __GI_abort () at abort.c:89 #2 0x00007ffff5149bd7 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x7fffd403b840 "snapshots.size() == 0", file=file@entry=0x7fffd40372a0 "./storage/rocksdb/rocksdb/utilities/transactions/write_prepared_txn_db.cc", line=line@entry=391, function=function@entry=0x7fffd403c700 <rocksdb::WritePreparedTxnDB::RollbackPrepared(unsigned long, unsigned long)::__PRETTY_FUNCTION__> "void rocksdb::WritePreparedTxnDB::RollbackPrepared(uint64_t, uint64_t)") at assert.c:92 #3 0x00007ffff5149c82 in __GI___assert_fail (assertion=assertion@entry=0x7fffd403b840 "snapshots.size() == 0", file=file@entry=0x7fffd40372a0 "./storage/rocksdb/rocksdb/utilities/transactions/write_prepared_txn_db.cc", line=line@entry=391, function=function@entry=0x7fffd403c700 <rocksdb::WritePreparedTxnDB::RollbackPrepared(unsigned long, unsigned long)::__PRETTY_FUNCTION__> "void rocksdb::WritePreparedTxnDB::RollbackPrepared(uint64_t, uint64_t)") at assert.c:101 #4 0x00007fffd3bc2777 in rocksdb::WritePreparedTxnDB::RollbackPrepared (this=0x62c000028200, prep_seq=59) at ./storage/rocksdb/rocksdb/utilities/transactions/write_prepared_txn_db.cc:391 #5 0x00007fffd3bb8ff7 in rocksdb::WritePreparedTxn::RollbackInternal (this=0x6160003ab080) at ./storage/rocksdb/rocksdb/utilities/transactions/write_prepared_txn.cc:356 #6 0x00007fffd3b628a6 in rocksdb::PessimisticTransaction::Rollback (this=0x6160003ab080) at ./storage/rocksdb/rocksdb/utilities/transactions/pessimistic_transaction.cc:341 #7 0x00007fffd3340b8a in myrocks::Rdb_transaction_impl::rollback (this=0x61400006f040) at ./storage/rocksdb/ha_rocksdb.cc:2454 #8 0x00007fffd32c9fde in myrocks::Rdb_transaction::commit (this=0x61400006f040) at ./storage/rocksdb/ha_rocksdb.cc:1970 #9 myrocks::rocksdb_commit (hton=<optimized out>, thd=<optimized out>, commit_tx=<optimized out>) at ./storage/rocksdb/ha_rocksdb.cc:3196 #10 0x0000000000beafb3 in ha_commit_low (thd=thd@entry=0x62600023d100, all=<optimized out>, run_after_commit=run_after_commit@entry=false) at ./sql/handler.cc:1807 #11 0x000000000284792b in MYSQL_BIN_LOG::process_commit_stage_queue (this=this@entry=0x65067c0 <mysql_bin_log>, thd=thd@entry=0x62600023d100, first=first@entry=0x62600023d100) at ./sql/binlog.cc:8268 #12 0x000000000286f6a6 in MYSQL_BIN_LOG::ordered_commit (this=this@entry=0x65067c0 <mysql_bin_log>, thd=thd@entry=0x62600023d100, all=all@entry=false, skip_commit=skip_commit@entry=false) at ./sql/binlog.cc:8887 #13 0x0000000002874532 in MYSQL_BIN_LOG::commit (this=<optimized out>, thd=0x62600023d100, all=<optimized out>) at ./sql/binlog.cc:8102 #14 0x0000000000bedd5b in ha_commit_trans (thd=thd@entry=0x62600023d100, all=all@entry=false, ignore_global_read_lock=ignore_global_read_lock@entry=false) at ./sql/handler.cc:1685 #15 0x00000000015271b5 in trans_commit_stmt (thd=thd@entry=0x62600023d100, ignore_global_read_lock=ignore_global_read_lock@entry=false) at ./sql/transaction.cc:479 #16 0x00000000012b195c in mysql_execute_command (thd=thd@entry=0x62600023d100, first_level=first_level@entry=true) at ./sql/sql_parse.cc:4357 #17 0x00000000012b3927 in mysql_parse (thd=thd@entry=0x62600023d100, parser_state=parser_state@entry=0x7fffc89b72f0) at ./sql/sql_parse.cc:4936 #18 0x00000000012b6dfa in dispatch_command (thd=thd@entry=0x62600023d100, com_data=com_data@entry=0x7fffc89b8b80, command=<optimized out>) at ./sql/sql_parse.cc:1589 #19 0x00000000012bcc11 in do_command (thd=thd@entry=0x62600023d100) at ./sql/sql_parse.cc:1214 #20 0x000000000158aba9 in handle_connection (arg=<optimized out>) at ./sql/conn_handler/connection_handler_per_thread.cc:308 #21 0x0000000003bd4fbd in pfs_spawn_thread (arg=0x6110002b36e0) at ./storage/perfschema/pfs.cc:2814 #22 0x00007ffff6c546ba in start_thread (arg=0x7fffc89b9700) at pthread_create.c:333 #23 0x00007ffff522341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109The crash can be repeated with the following mtr test:
--source include/have_rocksdb.inc --source include/count_sessions.inc # have_write_prepared if (`select count(*) = 0 from performance_schema.session_variables where variable_name = 'rocksdb_write_policy' and variable_value = 'write_prepared';`) { --skip Test requires write_prepared policy } # Create a table with a primary key and one secondary key as well as one # more column CREATE TABLE t1( a int ) ENGINE=ROCKSDB; --let $file = `SELECT CONCAT(@@datadir, "test_loadfile.txt")` --write_file $file 1 EOF --file_exists $file connect (other,localhost,root,,); start transaction with consistent snapshot; connection default; set session rocksdb_bulk_load=1; set session rocksdb_bulk_load_size=1; --disable_query_log --echo LOAD DATA INFILE <input_file> INTO TABLE t1; eval LOAD DATA INFILE '$file' INTO TABLE t1; --enable_query_log --remove_file $file set session rocksdb_bulk_load=0; # Cleanup disconnect other; DROP TABLE t1; --source include/wait_until_count_sessions.inc