Mysqld Crashed with Signal 6 on 5.7.29 version of Percona MySQL
Done
Description fields
General
Escalation
General
Escalation
Description
Hi Team,
We are experiencing a bug issue on MySQL 5.7.29 version , today one of our production mysql server crashed with signal 6 issue , please find the details below
TRANSACTION 63083864418, ACTIVE 0 sec mysql tables in use 1, locked 1 LOCK WAIT 3 lock struct(s), heap size 1136, 18 row lock(s) MySQL thread id 12231, OS thread handle 140230384146176, query id 8491032014 server name and IP mysqluser updating DELETE FROM table WHERE accountid=**** AND fileid = unhex('**************') AND deleteddate='2020-04-24 11:42:41' 2020-07-08T11:42:47.027708Z 12084 [Note] InnoDB: *** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 258 page no 772726 n bits 384 index GEN_CLUST_INDEX of table `database`.`table` trx id 63083864418 lock_mode X locks rec but not gap waiting 2020-07-08T11:42:47.027743Z 12084 [Note] InnoDB: *** (2) TRANSACTION:
TRANSACTION 63083864417, ACTIVE 0 sec updating or deleting, thread declared inside InnoDB 4999 mysql tables in use 1, locked 1 4 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 1 MySQL thread id 12084, OS thread handle 140230256154368, query id 8491032013 "hostname and IP here" mysqluser updating DELETE FROM table WHERE accountid=******* AND fileid = unhex('*********************') AND deleteddate='2020-04-24 11:42:41' 2020-07-08T11:42:47.027797Z 12084 [Note] InnoDB: *** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 258 page no 772726 n bits 384 index GEN_CLUST_INDEX of table `database`.`table` trx id 63083864417 lock_mode X locks rec but not gap 2020-07-08T11:42:47.027818Z 12084 [Note] InnoDB: *** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 258 page no 227170 n bits 1056 index recycle_bin_deleted_index of table `database`.`table` trx id 63083864417 lock_mode X locks rec but not gap waiting 2020-07-08T11:42:47.027840Z 12084 [Note] InnoDB: *** WE ROLL BACK TRANSACTION (1)
2020-07-08 11:42:52 0x7f89b2339700 InnoDB: Assertion failure in thread 140229376972544 in file rem0rec.cc line 586 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 11:42:53 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail. Please help us make Percona Server better by reporting any bugs at https://bugs.percona.com/
key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=1062 max_threads=4001 thread_count=575 connection_count=553 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1601081 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f8b8ce52000 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7f89b2338cf0 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xf18ceb] /usr/sbin/mysqld(handle_fatal_signal+0x471)[0xd50b01] /lib64/libpthread.so.0(+0xf630)[0x7fbbd37df630] /lib64/libc.so.6(gsignal+0x37)[0x7fbbd18dd387] /lib64/libc.so.6(abort+0x148)[0x7fbbd18dea78] /usr/sbin/mysqld[0x77c810] /usr/sbin/mysqld(_Z20rec_get_offsets_funcPKhPK12dict_index_tPmmPP16mem_block_info_t+0x56)[0x106db96] /usr/sbin/mysqld(_Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm+0x3528)[0x10d01e8] /usr/sbin/mysqld(_ZN11ha_innobase10index_readEPhPKhj16ha_rkey_function+0x355)[0xfaabb5] /usr/sbin/mysqld(_ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function+0x1cd)[0x7b622d]
Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7f8b8ce58030): DELETE FROM files_briefcase WHERE accountid = 5441534 LIMIT 100 Connection ID (thread ID): 63922 Status: NOT_KILLED
You may download the Percona Server operations manual by visiting http://www.percona.com/software/percona-server/. You may find information in the manual which will help you identify the cause of the crash.
Note: we have Upgraded our servers recently from 5.7.25 to 5.7.29.
Hi Team,
We are experiencing a bug issue on MySQL 5.7.29 version , today one of our production mysql server crashed with signal 6 issue , please find the details below
TRANSACTION 63083864418, ACTIVE 0 sec
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 1136, 18 row lock(s)
MySQL thread id 12231, OS thread handle 140230384146176, query id 8491032014 server name and IP mysqluser updating
DELETE FROM table WHERE accountid=**** AND fileid = unhex('**************') AND deleteddate='2020-04-24 11:42:41'
2020-07-08T11:42:47.027708Z 12084 [Note] InnoDB: *** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 258 page no 772726 n bits 384 index GEN_CLUST_INDEX of table `database`.`table` trx id 63083864418 lock_mode X locks rec but not gap waiting
2020-07-08T11:42:47.027743Z 12084 [Note] InnoDB: *** (2) TRANSACTION:
TRANSACTION 63083864417, ACTIVE 0 sec updating or deleting, thread declared inside InnoDB 4999
mysql tables in use 1, locked 1
4 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 1
MySQL thread id 12084, OS thread handle 140230256154368, query id 8491032013 "hostname and IP here" mysqluser updating
DELETE FROM table WHERE accountid=******* AND fileid = unhex('*********************') AND deleteddate='2020-04-24 11:42:41'
2020-07-08T11:42:47.027797Z 12084 [Note] InnoDB: *** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 258 page no 772726 n bits 384 index GEN_CLUST_INDEX of table `database`.`table` trx id 63083864417 lock_mode X locks rec but not gap
2020-07-08T11:42:47.027818Z 12084 [Note] InnoDB: *** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 258 page no 227170 n bits 1056 index recycle_bin_deleted_index of table `database`.`table` trx id 63083864417 lock_mode X locks rec but not gap waiting
2020-07-08T11:42:47.027840Z 12084 [Note] InnoDB: *** WE ROLL BACK TRANSACTION (1)
2020-07-08 11:42:52 0x7f89b2339700 InnoDB: Assertion failure in thread 140229376972544 in file rem0rec.cc line 586
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
11:42:53 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at https://bugs.percona.com/
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=1062
max_threads=4001
thread_count=575
connection_count=553
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1601081 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f8b8ce52000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f89b2338cf0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xf18ceb]
/usr/sbin/mysqld(handle_fatal_signal+0x471)[0xd50b01]
/lib64/libpthread.so.0(+0xf630)[0x7fbbd37df630]
/lib64/libc.so.6(gsignal+0x37)[0x7fbbd18dd387]
/lib64/libc.so.6(abort+0x148)[0x7fbbd18dea78]
/usr/sbin/mysqld[0x77c810]
/usr/sbin/mysqld(_Z20rec_get_offsets_funcPKhPK12dict_index_tPmmPP16mem_block_info_t+0x56)[0x106db96]
/usr/sbin/mysqld(_Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm+0x3528)[0x10d01e8]
/usr/sbin/mysqld(_ZN11ha_innobase10index_readEPhPKhj16ha_rkey_function+0x355)[0xfaabb5]
/usr/sbin/mysqld(_ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function+0x1cd)[0x7b622d]
/usr/sbin/mysqld(_ZN7handler16read_range_firstEPK12st_key_rangeS2_bb+0x5c)[0x7b6a5c]
/usr/sbin/mysqld(_ZN7handler21multi_range_read_nextEPPc+0x99)[0x7a9779]
/usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECT8get_nextEv+0x5a)[0xe0872a]
/usr/sbin/mysqld[0xbd11ca]
/usr/sbin/mysqld(_ZN14Sql_cmd_delete12mysql_deleteEP3THDy+0x10b6)[0xe43ea6]
/usr/sbin/mysqld(_ZN14Sql_cmd_delete7executeEP3THD+0xdf)[0xe446cf]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x255d)[0xc6e72d]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_stateb+0x475)[0xc73b75]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xb67)[0xc74817]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x1df)[0xc762df]
/usr/sbin/mysqld(handle_connection+0x2c0)[0xd376a0]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xf30fa4]
/lib64/libpthread.so.0(+0x7ea5)[0x7fbbd37d7ea5]
/lib64/libc.so.6(clone+0x6d)[0x7fbbd19a58dd]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f8b8ce58030): DELETE FROM files_briefcase WHERE accountid = 5441534 LIMIT 100
Connection ID (thread ID): 63922
Status: NOT_KILLED
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
Note: we have Upgraded our servers recently from 5.7.25 to 5.7.29.
Thanks and Regards
Jabir Baig Mirza