Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

LP #1441139: handle_fatal_signal (sig=11) in Queue<PROF_MEASUREMENT>::pop | sql/sql_profile.h:127 | handle_fatal_signal (sig=11) in my_lfind

Description

**Reported in Launchpad by Ramesh Sivaraman last update 18-08-2016 14:10:58

  •  

    • Testcase

DROP DATABASE test;CREATE DATABASE test;USE test;

INSTALL PLUGIN scalability_metrics SONAME 'scalability_metrics.so';
set session profiling=ON;
SET GLOBAL scalability_metrics_control=ON;
SET GLOBAL scalability_metrics_control=OFF;
SET GLOBAL scalability_metrics_control=ON;

The attached tarball gives the testcase as an exact match of our system,
including some handy utilities

$ vi {epoch}_mybase # Update base path in this file (the only change
required!). For non-binary (i.e. non-tarball/non-build-binary-distribution) distributions please update SOURCE_DIR location also.
$ ./{epoch}_init # Initializes the data dir
$ ./{epoch}_start # Starts mysqld
$ ./{epoch}_cl # To check mysqld is up
$ ./{epoch}_run # Run the testcase with pquery binary(produces
output)
$ vi /dev/shm/{epoch}/error.log.out # Verify the error log
$ ./{epoch}_gdb # Brings you to a gdb prompt attached to correct
mysqld
& generated core
$ ./{epoch}_parse_core # Create {epoch}_STD.gdb and {epoch}_FULL.gdb;
standard and full var gdb stack traces
etc.

  •  

    • GDB info

#0 0x00007ffce1b4a771 in __pthread_kill (threadid=<optimized out>, signo=11) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61

#1 0x0000000000abb47a in my_write_core (sig=11) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/mysys/stacktrace.c:422
#2 0x00000000007312c3 in handle_fatal_signal (sig=11) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/signal_handler.cc:236
#3 <signal handler called>
#4 0x000000000080bf58 in Queue<PROF_MEASUREMENT>::pop (this=0x7ffcc045e180) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/sql_profile.h:127
#5 0x0000000000809e91 in QUERY_PROFILE::~QUERY_PROFILE (this=0x7ffcc045e150, __in_chrg=<optimized out>) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/sql_profile.cc:295
#6 0x000000000080a17d in PROFILING::~PROFILING (this=0x7ffccff72be0, __in_chrg=<optimized out>) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/sql_profile.cc:349
#7 0x0000000000798db7 in THD::~THD (this=0x7ffccff71000, __in_chrg=<optimized out>) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/sql_class.cc:1783
#8 0x00000000007990a2 in THD::~THD (this=0x7ffccff71000, __in_chrg=<optimized out>) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/sql_class.cc:1840
#9 0x0000000000625aba in one_thread_per_connection_end (thd=0x7ffccff71000, block_pthread=true) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/mysqld.cc:2848
#10 0x00000000007a7dc4 in do_handle_one_connection (thd_arg=0x7ffccff71000) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/sql_connect.cc:1546
#11 0x00000000007a7815 in handle_one_connection (arg=0x7ffccff71000) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/sql_connect.cc:1443
#12 0x0000000000dcc8ac in pfs_spawn_thread (arg=0x7ffcdb346240) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/storage/perfschema/pfs.cc:1860
#13 0x00007ffce1b45df5 in start_thread (arg=0x7ffce2103700) at pthread_create.c:308
#14 0x00007ffce080f1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Environment

None

Activity

lpjirasync 
January 22, 2018 at 7:34 PM

**Comment from Launchpad by: Roel Van de Paar on: 01-06-2016 06:31:54

Also see bug 1587738

lpjirasync 
January 22, 2018 at 7:34 PM

**Comment from Launchpad by: Roel Van de Paar on: 01-06-2016 04:44:41

Another set of GLIBC crashes seen in combination with ON/OFF scalability_metrics_control

  •  

    •  

      • Error in `/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld': free(): invalid next size (fast): 0x00007f5f94026b30 ***
        ======= Backtrace: =========
        /lib64/libc.so.6(+0x7cfe1)[0x7f6042b6afe1]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld[0x1833f84]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(my_free+0x90)[0x1833d02]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld[0x153caeb]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z18plugin_unlock_listP3THDPPP13st_plugin_intm+0xce)[0x153cde0]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_ZN14Trans_delegate15before_rollbackEP3THDb+0x269)[0x14390cd]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z15ha_rollback_lowP3THDb+0x85)[0xf04a9e]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_ZN11TC_LOG_MMAP8rollbackEP3THDb+0x27)[0x1613b55]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z19trans_rollback_stmtP3THD+0x125)[0x1616885]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z21mysql_execute_commandP3THDb+0x7293)[0x1515922]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x685)[0x151707a]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xbf8)[0x150c035]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z10do_commandP3THD+0x520)[0x150af4f]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(handle_connection+0x2a0)[0x16447eb]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(pfs_spawn_thread+0x170)[0x1868024]
        /lib64/libpthread.so.0(+0x7dc5)[0x7f6044785dc5]
        /lib64/libc.so.6(clone+0x6d)[0x7f6042be421d]

/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld': corrupted double-linked list: 0x00007ffdb4026b20 ***

======= Backtrace: =========
/lib64/libc.so.6(+0x7b184)[0x7ffe642aa184]
/lib64/libc.so.6(+0x7d070)[0x7ffe642ac070]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld[0x1833f84]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(my_free+0x90)[0x1833d02]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld[0x1541fa4]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z21plugin_thdvar_cleanupP3THDb+0x18c)[0x1542154]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z14end_connectionP3THD+0x49)[0x14b933e]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(handle_connection+0x2c2)[0x164480d]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(pfs_spawn_thread+0x170)[0x1868024]
/lib64/libpthread.so.0(+0x7dc5)[0x7ffe65ec6dc5]
/lib64/libc.so.6(clone+0x6d)[0x7ffe6432521d]

  •  

    •  

      • Error in `/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld': corrupted double-linked list: 0x00007fb90c026b50 ***
        ======= Backtrace: =========
        /lib64/libc.so.6(+0x7bea2)[0x7fb9bd9e4ea2]
        /lib64/libc.so.6(+0x7de35)[0x7fb9bd9e6e35]
        /lib64/libc.so.6(__libc_malloc+0x4c)[0x7fb9bd9e887c]
        /lib64/libstdc++.so.6(_ZnwmRKSt9nothrow_t+0x1d)[0x7fb9be2a115d]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_ZN12Channel_info10create_thdEv+0x74)[0x1745984]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_ZN25Channel_info_local_socket10create_thdEv+0x1b)[0x1647855]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld[0x16443c7]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(handle_connection+0xe6)[0x1644631]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(pfs_spawn_thread+0x170)[0x1868024]
        /lib64/libpthread.so.0(+0x7dc5)[0x7fb9bf600dc5]
        /lib64/libc.so.6(clone+0x6d)[0x7fb9bda5f21d]

  •  

    •  

      • Error in `/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld': corrupted double-linked list: 0x00007fb13c026b50 ***
        ======= Backtrace: =========
        /lib64/libc.so.6(+0x7bea2)[0x7fb1ec9ffea2]
        /lib64/libc.so.6(+0x7de35)[0x7fb1eca01e35]
        /lib64/libc.so.6(__libc_malloc+0x4c)[0x7fb1eca0387c]
        /lib64/libstdc++.so.6(_ZnwmRKSt9nothrow_t+0x1d)[0x7fb1ed2bc15d]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_ZN12Channel_info10create_thdEv+0x74)[0x1745984]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_ZN25Channel_info_local_socket10create_thdEv+0x1b)[0x1647855]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld[0x16443c7]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(handle_connection+0xe6)[0x1644631]
        /sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(pfs_spawn_thread+0x170)[0x1868024]
        /lib64/libpthread.so.0(+0x7dc5)[0x7fb1ee61bdc5]
        /lib64/libc.so.6(clone+0x6d)[0x7fb1eca7a21d]

Produced with this testcase or variations thereof;

  1. mysqld options required for replay: --event-scheduler=ON --maximum-bulk_insert_buffer_size=1M --maximum-join_buffer_size=1M --maximum-max_heap_table_size=1M --maximum-max_join_size=1M --maximum-myisam_max_sort_file_size=1M --maximum-myisam_mmap_size=1M --maximum-myisam_sort_buffer_size=1M --maximum-optimizer_trace_max_mem_size=1M --maximum-preload_buffer_size=1M --maximum-query_alloc_block_size=1M --maximum-query_prealloc_size=1M --maximum-range_alloc_block_size=1M --maximum-read_buffer_size=1M --maximum-read_rnd_buffer_size=1M --maximum-sort_buffer_size=1M --maximum-tmp_table_size=1M --maximum-transaction_alloc_block_size=1M --maximum-transaction_prealloc_size=1M --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --plugin-load-add=audit_log=audit_log.so --plugin-load-add=tokudb=ha_tokudb.so --init-file=/home/roel/percona-qa/plugins.sql --tokudb-check-jemalloc=0 --innodb-api-bk-commit-interval=24 --performance-schema-digests-size=0 --tokudb-rpl-unique-checks-delay=12 --tokudb-directio=0 --tokudb-rpl-unique-checks-delay=24 --tokudb-auto-analyze=254 --tokudb_fanout=512
    DROP DATABASE transforms;CREATE DATABASE transforms;DROP DATABASE test;CREATE DATABASE test;USE test;
    show session variables like 'have_profiling';
    SET GLOBAL scalability_metrics_control=OFF;
    select count(*)FROM t1 where f2='';
    CREATE TABLE m3(c1 INT,c2 CHAR (1),c3 INT(1),c4 VARCHAR(1),c5 INT UNIQUE KEY,c6 NUMERIC(10,8) DEFAULT 3.141592);
    SET GLOBAL scalability_metrics_control=ON;
    CREATE TABLE m3(c1 INT,c2 CHAR (1),c3 INT(1),c4 VARCHAR(1),c5 INT UNIQUE KEY,c6 NUMERIC(10,8) DEFAULT 3.141592);

lpjirasync 
January 22, 2018 at 7:34 PM

**Comment from Launchpad by: Roel Van de Paar on: 28-05-2016 04:50:40

s/is/as/

lpjirasync 
January 22, 2018 at 7:34 PM

**Comment from Launchpad by: Roel Van de Paar on: 28-05-2016 04:43:08

Note that percona-qa is necessary for this testcase because the testcase has this in start:

1464402108]$ cat 1464402108_start | grep plugins.sql
...--init-file=/home/roel/percona-qa/plugins.sql...

Otherwise, use is always, see 1464402108_how_to_use.txt in tarball.

lpjirasync 
January 22, 2018 at 7:34 PM

**Comment from Launchpad by: Roel Van de Paar on: 28-05-2016 04:39:54

Ignore the testcase in #9. Got a good version which always works in terms of producing a mysqld crash. Uploading below. The GLIBC is not produced, but perhaps once in a while it could be (reduction was done on the GLIBC crash occurring, not mysqld crashing). It is highly sporadic, though regularly seen during overall QA trials. Then again; there are different GLIBC crashes. It is unclear if they are all related to the ON/OFF of scalability_metrics_control.

Done

Details

Assignee

Reporter

Labels

Priority

Created January 22, 2018 at 7:32 PM
Updated January 22, 2018 at 7:34 PM
Resolved January 22, 2018 at 7:33 PM