Updating auto increment value causes server to "go away" [ZN7myrocks10ha_rocksdb31update_auto_incr_val_from_fieldEv]
General
Escalation
General
Escalation
Description
Environment
None
Smart Checklist
Activity
Show:
George Lorch February 15, 2018 at 5:55 PM
These crashes are limited to auto increment columns that are of DOUBLE or FLOAT type only. Not at all a common use case. Particularly in a largely unused storage engine. Upstream already has responded to the issue and provided fixes.
roel.vandepaar February 15, 2018 at 8:53 AMEdited
These crashes are seen all through the runs (screenfillers). This basically breaks auto_increment as we know, is a serious bug, and it makes MyRocks un-GA imho. I am setting this to "blocker" as that is what it is IMHO. Could we un-cherry-pick the offending patch from the tree to make MyRocks stable again before release?
George Lorch February 12, 2018 at 4:57 PMEdited
Reproducible upstream on prod201704, likely introduced by one of these five commits :
NOPE https://github.com/facebook/mysql-5.6/commit/bb8183d38b401a0c24dcebb785d624ad34fe755f : Load autoincrement and hidden pk only once
NOPE https://github.com/facebook/mysql-5.6/commit/9f53fae746084823d35177b70a314d14c93a441b : Implement inplace alter for auto_increment
NOPE https://github.com/facebook/mysql-5.6/commit/3e68a8cd409ea63e296a5c788daec188d3a4b864 : Fix loading auto_increment values in tables with hidden pk
NOPE https://github.com/facebook/mysql-5.6/commit/b016d55a315277c0adf07d330bdeb9172f283b44 : Implement persistent auto_increment values
>>> this one >>> https://github.com/facebook/mysql-5.6/commit/4d66a663abfa60c6c1739d0612959b3457b5f2b1 : Fix auto_increment bugs with large values
(gdb) bt
#0 0x00007ffff5f53c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007ffff5f57028 in __GI_abort () at abort.c:89
#2 0x00000000012d98e1 in myrocks::rdb_get_int_col_max_value (field=0x7fffc8880448) at /ssd/percona/myrocks/facebook-mysql-5.6/storage/rocksdb/ha_rocksdb.cc:4702
#3 0x00000000012d9d75 in myrocks::ha_rocksdb::update_auto_incr_val_from_field (this=0x7fffc8820d10) at /ssd/percona/myrocks/facebook-mysql-5.6/storage/rocksdb/ha_rocksdb.cc:4815
#4 0x00000000012e6eae in myrocks::ha_rocksdb::update_pk (this=0x7fffc8820d10, kd=..., row_info=..., pk_changed=@0x7ffff7f7965b: false) at /ssd/percona/myrocks/facebook-mysql-5.6/storage/rocksdb/ha_rocksdb.cc:8946
#5 0x00000000012e76a3 in myrocks::ha_rocksdb::update_indexes (this=0x7fffc8820d10, row_info=..., pk_changed=@0x7ffff7f7965b: false) at /ssd/percona/myrocks/facebook-mysql-5.6/storage/rocksdb/ha_rocksdb.cc:9087
#6 0x00000000012e795d in myrocks::ha_rocksdb::update_write_row (this=0x7fffc8820d10, old_data=0x0, new_data=0x7fffc8880410 <incomplete sequence \375>, skip_unique_check=false)
at /ssd/percona/myrocks/facebook-mysql-5.6/storage/rocksdb/ha_rocksdb.cc:9159
#7 0x00000000012e5c1f in myrocks::ha_rocksdb::write_row (this=0x7fffc8820d10, buf=0x7fffc8880410 <incomplete sequence \375>) at /ssd/percona/myrocks/facebook-mysql-5.6/storage/rocksdb/ha_rocksdb.cc:8555
#8 0x0000000000d82ce3 in handler::ha_write_row (this=0x7fffc8820d10, buf=0x7fffc8880410 <incomplete sequence \375>) at /ssd/percona/myrocks/facebook-mysql-5.6/sql/handler.cc:7784
#9 0x0000000000f33745 in write_record (thd=0x7fffce9d1000, table=0x7fffc8864a00, info=0x7ffff7f79a50, update=0x7ffff7f79ad0) at /ssd/percona/myrocks/facebook-mysql-5.6/sql/sql_insert.cc:1889
#10 0x0000000000f314a9 in mysql_insert (thd=0x7fffce9d1000, table_list=0x7fffc8831100, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false)
at /ssd/percona/myrocks/facebook-mysql-5.6/sql/sql_insert.cc:1030
#11 0x0000000000f5a787 in mysql_execute_command (thd=0x7fffce9d1000, statement_start_time=0x7ffff7f7b208, post_parse=0x7ffff7f7b430) at /ssd/percona/myrocks/facebook-mysql-5.6/sql/sql_parse.cc:4526
#12 0x0000000000f6350a in mysql_parse (thd=0x7fffce9d1000, rawbuf=0x7fffc8831010 "insert into t1 values(2177,0)", length=29, parser_state=0x7ffff7f7bcd0, last_timer=0x7ffff7f7b430, async_commit=0x7ffff7f7b39b "")
at /ssd/percona/myrocks/facebook-mysql-5.6/sql/sql_parse.cc:7829
#13 0x0000000000f53bcd in dispatch_command (command=COM_QUERY, thd=0x7fffce9d1000, packet=0x7fffceaf0001 "insert into t1 values(2177,0)", packet_length=29, srv_session=0x0)
at /ssd/percona/myrocks/facebook-mysql-5.6/sql/sql_parse.cc:1897
#14 0x0000000000f51d2c in do_command (thd=0x7fffce9d1000) at /ssd/percona/myrocks/facebook-mysql-5.6/sql/sql_parse.cc:1212
#15 0x0000000000f0fb05 in do_handle_one_connection (thd_arg=0x7fffce9d1000) at /ssd/percona/myrocks/facebook-mysql-5.6/sql/sql_connect.cc:1142
#16 0x0000000000f0f51c in handle_one_connection (arg=0x7fffce9d1000) at /ssd/percona/myrocks/facebook-mysql-5.6/sql/sql_connect.cc:963
#17 0x0000000001afb299 in pfs_spawn_thread (arg=0x7ffff54fe100) at /ssd/percona/myrocks/facebook-mysql-5.6/storage/perfschema/pfs.cc:1860
#18 0x00007ffff6b0e184 in start_thread (arg=0x7ffff7f7d700) at pthread_create.c:312
#19 0x00007ffff601affd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111Hrvoje Matijakovic February 12, 2018 at 2:59 PMEdited
Found similar issue with different error string (ZN7myrocks10ha_rocksdb18get_auto_incrementEyyyPyS1):
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/bin/mysqld(my_print_stacktrace+0x35)[0x18715ac]
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/bin/mysqld(handle_fatal_signal+0x411)[0xe8ff98]
/lib64/libpthread.so.0(+0xf5e0)[0x7f41522d65e0]
/lib64/libc.so.6(gsignal+0x37)[0x7f41503df1f7]
/lib64/libc.so.6(abort+0x148)[0x7f41503e08e8]
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/lib/mysql/plugin/ha_rocksdb.so(+0x846c37)[0x7f411278ec37]
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/lib/mysql/plugin/ha_rocksdb.so(_ZN7myrocks10ha_rocksdb18get_auto_incrementEyyyPyS1_+0xb2)[0x7f41127a21ca]
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/bin/mysqld(_ZN7handler21update_auto_incrementEv+0x375)[0xf24c8b]
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/lib/mysql/plugin/ha_rocksdb.so(_ZN7myrocks10ha_rocksdb9write_rowEPh+0xf7)[0x7f411279a973]
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/bin/mysqld(_ZN7handler12ha_write_rowEPh+0x264)[0xf2e9a8]
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/bin/mysqld(_Z12write_recordP3THDP5TABLEP9COPY_INFOS4_+0xe79)[0x1757a92]
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/bin/mysqld(_ZN14Sql_cmd_insert12mysql_insertEP3THDP10TABLE_LIST+0xe4d)[0x1754bf3]
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/bin/mysqld(_ZN14Sql_cmd_insert7executeEP3THD+0x181)[0x175b595]
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/bin/mysqld(_Z21mysql_execute_commandP3THDb+0x2dd3)[0x153541c]
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/bin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x705)[0x153b3c3]
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/bin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xc59)[0x152ff13]
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/bin/mysqld(_Z10do_commandP3THD+0x521)[0x152ed9b]
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/bin/mysqld(handle_connection+0x294)[0x166f4c8]
/sda/PS090218-percona-server-5.7.21-20-linux-x86_64-debug/bin/mysqld(pfs_spawn_thread+0x170)[0x191b351]
/lib64/libpthread.so.0(+0x7e25)[0x7f41522cee25]
/lib64/libc.so.6(clone+0x6d)[0x7f41504a234d]Test case:
# mysqld options required for replay: --plugin-load-add=RocksDB=ha_rocksdb.so
CREATE DATABASE test;
USE test;
CREATE TABLE t1(C1 FLOAT AUTO_INCREMENT KEY,C2 CHAR)ENGINE=RocksDB;
INSERT DELAYED INTO t1 VALUES(0,0);Done
Details
Details
Assignee
George Lorch
George Lorch(Deactivated)Reporter
Hrvoje Matijakovic
Hrvoje MatijakovicTime tracking
4h 3m logged
Components
Fix versions
Affects versions
Priority
HighSmart Checklist
Open Smart Checklist
Smart Checklist
Open Smart Checklist
Created February 12, 2018 at 11:02 AM
Updated March 6, 2024 at 1:31 PM
Resolved February 16, 2018 at 3:41 AM
gdb info
(gdb) bt +bt #0 0x00007f9e1b0499b1 in __pthread_kill (threadid=<optimized out>, signo=6) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61 #1 0x0000000001871633 in my_write_core (sig=6) at /home/hrvoje/worktable/PS-5.7.21_dbg/mysys/stacktrace.c:249 #2 0x0000000000e9018f in handle_fatal_signal (sig=6) at /home/hrvoje/worktable/PS-5.7.21_dbg/sql/signal_handler.cc:223 #3 <signal handler called> #4 0x00007f9e191551f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #5 0x00007f9e191568e8 in __GI_abort () at abort.c:90 #6 0x00007f9ddb38ec37 in myrocks::rdb_get_int_col_max_value (field=0x7f9dd14b0868) at /home/hrvoje/worktable/PS-5.7.21_dbg/storage/rocksdb/ha_rocksdb.cc:4389 #7 0x00007f9ddb38f09c in myrocks::ha_rocksdb::update_auto_incr_val_from_field (this=0x7f9dd1454030) at /home/hrvoje/worktable/PS-5.7.21_dbg/storage/rocksdb/ha_rocksdb.cc:4502 #8 0x00007f9ddb39bceb in myrocks::ha_rocksdb::update_pk (this=0x7f9dd1454030, kd=..., row_info=..., pk_changed=@0x7f9e1b638b5f: false) at /home/hrvoje/worktable/PS-5.7.21_dbg/storage/rocksdb/ha_rocksdb.cc:8606 #9 0x00007f9ddb39c4ff in myrocks::ha_rocksdb::update_indexes (this=0x7f9dd1454030, row_info=..., pk_changed=@0x7f9e1b638b5f: false) at /home/hrvoje/worktable/PS-5.7.21_dbg/storage/rocksdb/ha_rocksdb.cc:8735 #10 0x00007f9ddb39c7c3 in myrocks::ha_rocksdb::update_write_row (this=0x7f9dd1454030, old_data=0x0, new_data=0x7f9dd14b0830 <incomplete sequence \375>, skip_unique_check=false) at /home/hrvoje/worktable/PS-5.7.21_dbg/storage/rocksdb/ha_rocksdb.cc:8807 #11 0x00007f9ddb39a9bb in myrocks::ha_rocksdb::write_row (this=0x7f9dd1454030, buf=0x7f9dd14b0830 <incomplete sequence \375>) at /home/hrvoje/worktable/PS-5.7.21_dbg/storage/rocksdb/ha_rocksdb.cc:8221 #12 0x0000000000f2e9a8 in handler::ha_write_row (this=0x7f9dd1454030, buf=0x7f9dd14b0830 <incomplete sequence \375>) at /home/hrvoje/worktable/PS-5.7.21_dbg/sql/handler.cc:8452 #13 0x0000000001757a92 in write_record (thd=0x7f9dd1419000, table=0x7f9dd147b220, info=0x7f9e1b638ec0, update=0x7f9e1b638f40) at /home/hrvoje/worktable/PS-5.7.21_dbg/sql/sql_insert.cc:1877 #14 0x0000000001754bf3 in Sql_cmd_insert::mysql_insert (this=0x7f9dd142d1c0, thd=0x7f9dd1419000, table_list=0x7f9dd142cc30) at /home/hrvoje/worktable/PS-5.7.21_dbg/sql/sql_insert.cc:769 #15 0x000000000175b595 in Sql_cmd_insert::execute (this=0x7f9dd142d1c0, thd=0x7f9dd1419000) at /home/hrvoje/worktable/PS-5.7.21_dbg/sql/sql_insert.cc:3119 #16 0x000000000153541c in mysql_execute_command (thd=0x7f9dd1419000, first_level=true) at /home/hrvoje/worktable/PS-5.7.21_dbg/sql/sql_parse.cc:3746 #17 0x000000000153b3c3 in mysql_parse (thd=0x7f9dd1419000, parser_state=0x7f9e1b63a470) at /home/hrvoje/worktable/PS-5.7.21_dbg/sql/sql_parse.cc:5901 #18 0x000000000152ff13 in dispatch_command (thd=0x7f9dd1419000, com_data=0x7f9e1b63ac50, command=COM_QUERY) at /home/hrvoje/worktable/PS-5.7.21_dbg/sql/sql_parse.cc:1490 #19 0x000000000152ed9b in do_command (thd=0x7f9dd1419000) at /home/hrvoje/worktable/PS-5.7.21_dbg/sql/sql_parse.cc:1021 #20 0x000000000166f4c8 in handle_connection (arg=0x7f9e01246850) at /home/hrvoje/worktable/PS-5.7.21_dbg/sql/conn_handler/connection_handler_per_thread.cc:312 #21 0x000000000191b351 in pfs_spawn_thread (arg=0x7f9e17018d20) at /home/hrvoje/worktable/PS-5.7.21_dbg/storage/perfschema/pfs.cc:2190 #22 0x00007f9e1b044e25 in start_thread (arg=0x7f9e1b63b700) at pthread_create.c:308 #23 0x00007f9e1921834d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113Testcase:
CREATE DATABASE test; USE test; CREATE TABLE t1(C1 DOUBLE AUTO_INCREMENT KEY,C2 CHAR)ENGINE=RocksDB; insert into t1 values(2177,0);Version (release branch for RM-321):
Server version: 5.7.21-20-debug MySQL Community Server (GPL)