LP #1632600: handle_fatal_signal (sig=11) in my_lfind

Description

**Reported in Launchpad by Roel Van de Paar last update 12-10-2016 08:45:54

This is a multi-threaded bug. It was reduced down to this small testcase;

  1. mysqld options required for replay: --maximum-tmp_table_size=1M --maximum-transaction_prealloc_size=1M --sql_mode=ONLY_FULL_GROUP_BY --thread_handling=pool-of-threads --log-bin=binlog --plugin-load-add=tokudb=ha_tokudb.so --init-file=/home/roel/percona-qa/plugins_56.sql --tokudb-check-jemalloc=0 --performance-schema-events-waits-history-size=0 --tokudb-last-lock-timeout=4
    DROP DATABASE transforms;
    CREATE TABLE t1(id INT,a BLOB COLUMN_FORMAT COMPRESSED WITH COMPRESSION_DICTIONARY numbers)ENGINE=TokuDB;

And it then needs a multi-thread wrapper (like the one in percona-qa perhaps, or reducer) to run it

Laurynas analyzed the stacks and the issue looks to be a race in threadpool.

Thread 0:

Core was generated by `/sda/COMP7-PS111016-percona-server-5.6.32-78.0-linux-x86_64-debug/bin/mysqld --'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000000000e4d32d in my_lfind (head=0x7f6e2c438e28, cs=0x14e9260 <my_charset_bin>, hashnr=674812401, key=0x7f6e56c5e050 "\331\061\242$\336\255]\r'K8\001(\372test", keylen=212, cursor=0x7f6e56c5dfb0, pins=0x7f6e2d816300) at /git/compression7_dbg/mysys/lf_hash.c:95
95 link= cursor->curr->link;
(gdb) bt
+bt
#0 0x0000000000e4d32d in my_lfind (head=0x7f6e2c438e28, cs=0x14e9260 <my_charset_bin>, hashnr=674812401, key=0x7f6e56c5e050 "\331\061\242$\336\255]\r'K8\001(\372test", keylen=212, cursor=0x7f6e56c5dfb0, pins=0x7f6e2d816300) at /git/compression7_dbg/mysys/lf_hash.c:95
#1 0x0000000000e4d7ab in my_lsearch (head=0x7f6e2c438e28, cs=0x14e9260 <my_charset_bin>, hashnr=674812401, key=0x7f6e56c5e050 "\331\061\242$\336\255]\r'K8\001(\372test", keylen=212, pins=0x7f6e2d816300) at /git/compression7_dbg/mysys/lf_hash.c:267
#2 0x0000000000e4de4e in lf_hash_search (hash=0x17d5780 <digest_hash>, pins=0x7f6e2d816300, key=0x7f6e56c5e050, keylen=212) at /git/compression7_dbg/mysys/lf_hash.c:462
#3 0x0000000000ae1956 in find_or_create_digest (thread=0x7f6e6b9d5700, digest_storage=0x7f6e4ffc4c60, schema_name=0x7f6e4ffc4d48 "test", schema_name_length=4) at /git/compression7_dbg/storage/perfschema/pfs_digest.cc:223
#4 0x0000000000b1696d in end_statement_v1 (locker=0x7f6e4ffc4ca0, stmt_da=0x7f6e4ffc64a0) at /git/compression7_dbg/storage/perfschema/pfs.cc:4815
#5 0x00000000007e3239 in inline_mysql_end_statement (locker=0x7f6e4ffc4ca0, stmt_da=0x7f6e4ffc64a0) at /git/compression7_dbg/include/mysql/psi/mysql_statement.h:215
#6 0x00000000007e8065 in dispatch_command (command=COM_QUERY, thd=0x7f6e4ffc3000, packet=0x7f6e4ffc7001 "", packet_length=87) at /git/compression7_dbg/sql/sql_parse.cc:1911
#7 0x00000000007e595d in do_command (thd=0x7f6e4ffc3000) at /git/compression7_dbg/sql/sql_parse.cc:1062
#8 0x00000000008cbba0 in threadpool_process_request (thd=0x7f6e4ffc3000) at /git/compression7_dbg/sql/threadpool_common.cc:314
#9 0x00000000008ce48a in handle_event (connection=0x7f6e4fef6bb0) at /git/compression7_dbg/sql/threadpool_unix.cc:1564
#10 0x00000000008ce6e4 in worker_main (param=0x17bdc00 <all_groups+512>) at /git/compression7_dbg/sql/threadpool_unix.cc:1617
#11 0x0000000000b10701 in pfs_spawn_thread (arg=0x7f6e2a084020) at /git/compression7_dbg/storage/perfschema/pfs.cc:1860
#12 0x00007f6e8078edc5 in start_thread (arg=0x7f6e56c5f700) at pthread_create.c:308
#13 0x00007f6e7ebebced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Signal thread:

Thread 69 (Thread 0x7f6e5fc6b700 (LWP 4988)):
+bt
#0 0x00007f6e7eec86e3 in _Unwind_IteratePhdrCallback (info=info@entry=0x7f6e5fc69050, size=size@entry=64, ptr=ptr@entry=0x7f6e5fc690d0) at ../../../libgcc/unwind-dw2-fde-dip.c:398
#1 0x00007f6e7ec25fac in _GI__dl_iterate_phdr (callback=callback@entry=0x7f6e7eec82e0 <_Unwind_IteratePhdrCallback>, data=data@entry=0x7f6e5fc690d0) at dl-iteratephdr.c:76
#2 0x00007f6e7eec8c1f in Unwind_Find_FDE (pc=0x7f6e7ebffbe5 <GI__backtrace+85>, bases=bases@entry=0x7f6e5fc69268) at ../../../libgcc/unwind-dw2-fde-dip.c:461
#3 0x00007f6e7eec5d8c in uw_frame_state_for (context=context@entry=0x7f6e5fc691c0, fs=fs@entry=0x7f6e5fc692b0) at ../../../libgcc/unwind-dw2.c:1245
#4 0x00007f6e7eec7019 in _Unwind_Backtrace (trace=0x7f6e7ebffa70 <backtrace_helper>, trace_argument=0x7f6e5fc69470) at ../../../libgcc/unwind.inc:290
#5 0x00007f6e7ebffbe6 in _GI__backtrace (array=<optimized out>, size=128) at ../sysdeps/x86_64/backtrace.c:109
#6 0x0000000000ac1b34 in my_print_stacktrace (stack_bottom=0x7f6e5fc6ac08 "", thread_stack=262144) at /git/compression7_dbg/mysys/stacktrace.c:224
#7 0x00000000007361e6 in handle_fatal_signal (sig=11) at /git/compression7_dbg/sql/signal_handler.cc:163
#8 <signal handler called>
#9 0x0000000000e4d32d in my_lfind (head=0x7f6e2d83f528, cs=0x14e9260 <my_charset_bin>, hashnr=694946247, key=0x7f6e5fc6a050 "O(r\332\353\317m\352\207\032u\t\324\354compressed_db", keylen=212, cursor=0x7f6e5fc69fb0, pins=0x7f6e2d816280) at /git/compression7_dbg/mysys/lf_hash.c:95
#10 0x0000000000e4d7ab in my_lsearch (head=0x7f6e2d83f528, cs=0x14e9260 <my_charset_bin>, hashnr=694946247, key=0x7f6e5fc6a050 "O(r\332\353\317m\352\207\032u\t\324\354compressed_db", keylen=212, pins=0x7f6e2d816280) at /git/compression7_dbg/mysys/lf_hash.c:267
#11 0x0000000000e4de4e in lf_hash_search (hash=0x17d5780 <digest_hash>, pins=0x7f6e2d816280, key=0x7f6e5fc6a050, keylen=212) at /git/compression7_dbg/mysys/lf_hash.c:462
#12 0x0000000000ae1956 in find_or_create_digest (thread=0x7f6e6b9d3600, digest_storage=0x7f6e4ffb2c60, schema_name=0x7f6e4ffb2d48 "compressed_db", schema_name_length=13) at /git/compression7_dbg/storage/perfschema/pfs_digest.cc:223
#13 0x0000000000b1696d in end_statement_v1 (locker=0x7f6e4ffb2ca0, stmt_da=0x7f6e4ffb44a0) at /git/compression7_dbg/storage/perfschema/pfs.cc:4815
#14 0x00000000007e3239 in inline_mysql_end_statement (locker=0x7f6e4ffb2ca0, stmt_da=0x7f6e4ffb44a0) at /git/compression7_dbg/include/mysql/psi/mysql_statement.h:215
#15 0x00000000007e8065 in dispatch_command (command=COM_QUERY, thd=0x7f6e4ffb1000, packet=0x7f6e4ffb5001 "", packet_length=65) at /git/compression7_dbg/sql/sql_parse.cc:1911
#16 0x00000000007e595d in do_command (thd=0x7f6e4ffb1000) at /git/compression7_dbg/sql/sql_parse.cc:1062
#17 0x00000000008cbba0 in threadpool_process_request (thd=0x7f6e4ffb1000) at /git/compression7_dbg/sql/threadpool_common.cc:314
#18 0x00000000008ce48a in handle_event (connection=0x7f6e4fef6af0) at /git/compression7_dbg/sql/threadpool_unix.cc:1564
#19 0x00000000008ce6e4 in worker_main (param=0x17be800 <all_groups+3584>) at /git/compression7_dbg/sql/threadpool_unix.cc:1617
#20 0x0000000000b10701 in pfs_spawn_thread (arg=0x7f6e2d01b0c0) at /git/compression7_dbg/storage/perfschema/pfs.cc:1860
#21 0x00007f6e8078edc5 in start_thread (arg=0x7f6e5fc6b700) at pthread_create.c:308
#22 0x00007f6e7ebebced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Upon entering gdb;

(gdb) p cursor
+p cursor
$1 = (CURSOR *) 0x7f6e56c5dfb0

(gdb) p cursor->curr
+p cursor->curr
$2 = (LF_SLIST *) 0x2f62645f64657372

All details will be uploaded.

Environment

None

Smart Checklist

Activity

Show:

Kathy Williamson April 22, 2020 at 4:09 PM

MySQL 5.6 is scheduled for EOL in Feb 2021.  At this time, we do not believe that this issue will be fixed prior to EOL.  If you believe this issue is important enough to be be fixed prior to EOL, or that it also affects a later version, please leave a comment and we will consider the new information

lpjirasync January 24, 2018 at 10:32 AM

**Comment from Launchpad by: Roel Van de Paar on: 12-10-2016 08:35:17

Adding core dump + mysqld + ldd files + full data dir and original thread sql files etc.

To init gdb:

gdb ./mysqld/mysqld ./data/core.4840.1000.1000.11.1476222798.mysqld

1037.sql_out_out is the final testcase (listed above)

Even has reducer1037.sh included which can be path-modified to reproduce in-spe

Won't Do

Details

Assignee

Reporter

Affects versions

Priority

Smart Checklist

Created January 24, 2018 at 10:32 AM
Updated April 22, 2020 at 4:09 PM
Resolved April 22, 2020 at 4:09 PM