Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Percona Server 5.6.40 crashing in function row_search_for_mysql(

Description

The issue sometimes happens with a DELETE, sometimes with a SELECT, other times with CHECK TABLE.

 

This sometimes fails with signal 6, other times with signal 11

 

Stack trace:

#0 0x00007f1c037d2a01 in pthread_kill () from /home/carlos.tutte/.dwz/Percona-XtraDB-Cluster-56-5.6.40-26.25.1.el7.x86_64/libpthread.so.0 #1 0x000000000068061d in handle_fatal_signal (sig=11) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/signal_handler.cc:248 #2 <signal handler called> #3 0x00007f1c0192117d in _int_free () from /home/carlos.tutte/.dwz/Percona-XtraDB-Cluster-56-5.6.40-26.25.1.el7.x86_64/libc.so.6 #4 0x0000000000a898c2 in row_search_for_mysql (buf=buf@entry=0x7f15a0015be0 "", mode=<optimized out>, mode@entry=0, prebuilt=0x7f15a00115c8, match_mode=match_mode@entry=0, direction=direction@entry=1) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/storage/innobase/row/row0sel.cc:5519 #5 0x00000000009db609 in general_fetch (match_mode=0, direction=1, buf=0x7f15a0015be0 "", this=0x7f15a0fc24a0) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/storage/innobase/handler/ha_innodb.cc:9871 #6 ha_innobase::index_next (this=0x7f15a0fc24a0, buf=0x7f15a0015be0 "") at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/storage/innobase/handler/ha_innodb.cc:9938 #7 0x00000000005bd5cc in handler::ha_index_next (this=0x7f15a0fc24a0, buf=0x7f15a0015be0 "") at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/handler.cc:3194 #8 0x00000000005c19c0 in handler::read_range_next (this=0x7f15a0fc24a0) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/handler.cc:7345 #9 0x0000000000ba1e59 in ha_partition::handle_unordered_next (this=0x7f15a099f110, buf=0x7f15a0015be0 "", is_next_same=<optimized out>) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/ha_partition.cc:5940 #10 0x00000000005b8b22 in handler::multi_range_read_next (this=0x7f15a099f110, range_info=0x7f15f95244f0) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/handler.cc:6431 #11 0x000000000081ceda in QUICK_RANGE_SELECT::get_next (this=0x7f154c7c3de0) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/opt_range.cc:10644 #12 0x0000000000840d1d in rr_quick (info=0x7f154c774e48) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/records.cc:369 #13 0x00000000006e5041 in sub_select (join=0x7f154c773520, join_tab=0x7f154c774db8, end_of_records=<optimized out>) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/sql_executor.cc:1262 #14 0x00000000006e4388 in do_select (join=0x7f154c773520) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/sql_executor.cc:936 #15 JOIN::exec (this=0x7f154c773520) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/sql_executor.cc:194 #16 0x0000000000733645 in mysql_execute_select (free_join=true, select_lex=0x25c6b28, thd=0x25c4000) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/sql_select.cc:1101 #17 mysql_select (thd=thd@entry=0x25c4000, tables=0x7f154c73a7a0, wild_num=0, fields=..., conds=0x7f154c73b470, order=order@entry=0x25c6cf0, group=group@entry=0x25c6c28, having=0x0, select_options=2147748609, result=result@entry=0x7f154c73bfc0, unit=unit@entry=0x25c64e0, select_lex=select_lex@entry=0x25c6b28) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/sql_select.cc:1222 #18 0x0000000000733ed5 in handle_select (thd=thd@entry=0x25c4000, result=result@entry=0x7f154c73bfc0, setup_tables_done_option=setup_tables_done_option@entry=0) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/sql_select.cc:110 #19 0x000000000056399c in execute_sqlcom_select (thd=thd@entry=0x25c4000, all_tables=<optimized out>) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/sql_parse.cc:6400 #20 0x0000000000708535 in mysql_execute_command (thd=thd@entry=0x25c4000) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/sql_parse.cc:3411 #21 0x000000000070f858 in mysql_parse (thd=thd@entry=0x25c4000, rawbuf=rawbuf@entry=0x7f154c73a2a0 "SELECT\n\t\t\t\tDISTINCT LOWER(bdr) bdr\n\t\t\tFROM\n\t\t\t\ttrace.dr_tracetx\n\t\t\tWHERE\n\t\t\t\ttp = 21\n\t\t\t\tAND bc IN ('9848u69e91q0r','crsqrmpe91r05')\n\t\t\t\tAND ts BETWEEN FROM_UNIXTIME(1553000907)\n\t\t\t\t\tAND DATE_ADD(FROM_UNIXTIME(1553000907),INTERVAL 24 HOUR)", length=length@entry=239, parser_state=parser_state@entry=0x7f15f95261b0) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/sql_parse.cc:7868 #22 0x0000000000710129 in wsrep_mysql_parse (thd=thd@entry=0x25c4000, rawbuf=0x7f154c73a2a0 "SELECT\n\t\t\t\tDISTINCT LOWER(bdr) bdr\n\t\t\tFROM\n\t\t\t\ttrace.dr_tracetx\n\t\t\tWHERE\n\t\t\t\ttp = 21\n\t\t\t\tAND bc IN ('9848u69e91q0r','crsqrmpe91r05')\n\t\t\t\tAND ts BETWEEN FROM_UNIXTIME(1553000907)\n\t\t\t\t\tAND DATE_ADD(FROM_UNIXTIME(1553000907),INTERVAL 24 HOUR)", length=239, parser_state=parser_state@entry=0x7f15f95261b0) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/sql_parse.cc:7560 #23 0x0000000000712c04 in dispatch_command (command=<optimized out>, command@entry=COM_QUERY, thd=thd@entry=0x25c4000, packet=packet@entry=0x25c7a21 "", packet_length=packet_length@entry=246) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/sql_parse.cc:1742 #24 0x00000000007142f9 in do_command (thd=0x25c4000) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/sql_parse.cc:1187 #25 0x00000000006dacf2 in do_handle_one_connection (thd_arg=thd_arg@entry=0x1f38d40) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/sql_connect.cc:1620 #26 0x00000000006daf20 in handle_one_connection (arg=arg@entry=0x1f38d40) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/sql/sql_connect.cc:1517 #27 0x00000000009506a6 in pfs_spawn_thread (arg=0x1f415b0) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/storage/perfschema/pfs.cc:1861 #28 0x00007f1c037cde25 in start_thread () from /home/carlos.tutte/.dwz/Percona-XtraDB-Cluster-56-5.6.40-26.25.1.el7.x86_64/libpthread.so.0 #29 0x00007f1c0199ebad in clone () from /home/carlos.tutte/.dwz/Percona-XtraDB-Cluster-56-5.6.40-26.25.1.el7.x86_64/libc.so.6

The software failing is PXC but the issue is on PS.

 

Segmentation fault is happening at frame 3 

#3  0x00007f1c0192117d in _int_free () from /home/carlos.tutte/.dwz/Percona-XtraDB-Cluster-56-5.6.40-26.25.1.el7.x86_64/libc.so.6

Which is called from:

#4  0x0000000000a898c2 in row_search_for_mysql (buf=buf@entry=0x7f15a0015be0 "", mode=<optimized out>, mode@entry=0, prebuilt=0x7f15a00115c8, match_mode=match_mode@entry=0, direction=direction@entry=1) at /usr/src/debug/Percona-XtraDB-Cluster-5.6.40-84.0/storage/innobase/row/row0sel.cc:5519

 

Crash happening on line 5519:

(gdb) list +list 5514    5515    func_exit: 5516  trx->op_info = ""; 5517    5518  if (end_range_cache != NULL) 5519      ut_free(end_range_cache); 5520  } 5521    5522  if (UNIV_LIKELY_NULL(heap)) { 5523      mem_heap_free(heap); +p end_range_cache $1 = (unsigned char *) 0x7f154c7ccf20 "\370\255\332\061b8he4XjoX9icgQJpx5nIJ.0cR.LEb\203"

 

 

Environment

None

AFFECTED CS IDs

https://percona.zendesk.com/agent/tickets/249858 and it's followups: https://percona.zendesk.com/agent/tickets/246134 https://percona.zendesk.com/agent/tickets/240684 https://percona.zendesk.com/agent/tickets/254518

Attachments

2
  • 22 Apr 2019, 08:05 AM
  • 19 Apr 2019, 08:47 AM

Smart Checklist

Activity

Show:

Kathy Williamson April 22, 2020 at 3:43 PM

MySQL 5.6 is scheduled for EOL in Feb 2021.  At this time, we do not believe that this issue will be fixed prior to EOL.  If you believe this issue is important enough to be be fixed prior to EOL, or that it also affects a later version, please leave a comment and we will consider the new information

Satya Bodapati July 9, 2019 at 8:09 AM
Edited

From mysqld.core.10001.31711

the rec belongs to space_id: 5263, page number: 662865

{name = 0x3411ac8 "trace/dr_tracetx#P#w0", id = 5623

tablespace size = 670208 pages = 670208 * 8K = 5.11328125 GB (expected size)

Are we sure we got right tablespace? The size from uploaded datafiles is
-rwxrwxrwx. 1 lalit.choudhary 10077 4.3G May 30 00:42 /bigdisk/lalit/254518_test/trace.dr_tracetx/dr_tracetx#P#w0.ibd

Can we get the right ibd file? (ibd file just after crash)

Satya Bodapati July 4, 2019 at 2:03 PM

End of the day report:
The pattern from both cores (rec_get_offsets() assert) failure is that

1. record header is corrupted in both cases. The allowed values for record type are
#define REC_STATUS_ORDINARY 0
#define REC_STATUS_NODE_PTR 1
#define REC_STATUS_INFIMUM 2
#define REC_STATUS_SUPREMUM 3

but we see values 4 & 7 and Hence it crashes(asserts).

2. Pages are possibly newly created. The checksum and LSN is both zero.

It is not clear whether these pages exist on disk. The IBD file seems smaller than the page number from core.

Satya Bodapati July 4, 2019 at 12:12 PM

[ the absence of lower stack frames hampers the analysis

Satya Bodapati July 4, 2019 at 12:11 PM

Can you find out why we get symbol table issue now? Even the first core (originally used carlos) doesn't give proper stack trace ( with function parameters and values).

Won't Do

Details

Assignee

Reporter

Time tracking

3d 4h logged

Affects versions

Priority

Smart Checklist

Created March 28, 2019 at 4:45 PM
Updated March 6, 2024 at 12:14 PM
Resolved April 22, 2020 at 3:43 PM