prepare crashed when tmpdir in datadir

Description

os:redhat 6.8,mysql:5.6.23,xtrabackup:2.4.16

 

import my.cnf configuration:

datadir=/data/owsdb/owswebdb

tmpdir=/data/owsdb/owswebdb/tmpdir

we can see that tmpdir is in datadir,and there always are some tmp disk table files in tmpdir.

 

when I use xtrabackup to make a full backup, it's ok.

then I use xtrabackup to do prepare step,the xtrabackup crash.

here is the log:

xtrabackup: recognized server arguments: --innodb_checksum_algorithm=innodb --innodb_log_checksum_algorithm=innodb --innodb_data_file_path=ibdata1:12M:autoextend --innodb_log_files_in_group=2 --innodb_log_file_size=524288000 --innodb_fast_checksum=0 --innodb_page_size=16384 --innodb_log_block_size=512 --innodb_undo_directory=. --innodb_undo_tablespaces=0 --server-id=0 --redo-log-version=0
xtrabackup: recognized client arguments: --prepare=1 --target-dir=./tmp/
/home/owsdb/xtrabackup/bin/xtrabackup version 2.4.16 based on MySQL server 5.7.26 Linux (x86_64) (revision id: c807cfa)
xtrabackup: cd to /data/backup/owswebdb/tmp/
xtrabackup: This target seems to be not prepared yet.
InnoDB: Number of pools: 1
xtrabackup: xtrabackup_logfile detected: size=8388608, start_lsn=(496416293325)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = .
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = .
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 8388608
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = .
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = .
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 8388608
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: PUNCH HOLE support available
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Uses event mutexes
InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
InnoDB: Compressed tables use zlib 1.2.3
InnoDB: Number of pools: 1
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M
InnoDB: Completed initialization of buffer pool
InnoDB: page_cleaner coordinator priority: -20
InnoDB: Highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 496416293325
InnoDB: Doing recovery: scanned up to log sequence number 496419657312 (45%)
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Removing missing table `tmpdir/#sql331a_256d9de_38` from InnoDB data dictionary.
InnoDB: Removing missing table `tmpdir/#sql331a_256f843_c` from InnoDB data dictionary.
InnoDB: Removing missing table `tmpdir/#sql331a_256fceb_1d` from InnoDB data dictionary.
InnoDB: Removing missing table `tmpdir/#sql331a_256ffad_18` from InnoDB data dictionary.
InnoDB: Removing missing table `tmpdir/#sql331a_256ffd0_19` from InnoDB data dictionary.
InnoDB: Removing missing table `tmpdir/#sql331a_2570131_24` from InnoDB data dictionary.
InnoDB: Removing missing table `tmpdir/#sql331a_25709f3_5` from InnoDB data dictionary.
InnoDB: Removing missing table `tmpdir/#sql331a_25709fe_8` from InnoDB data dictionary.
InnoDB: Removing missing table `tmpdir/#sql331a_2570a06_4` from InnoDB data dictionary.
InnoDB: Creating shared tablespace for temporary tables
InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
InnoDB: File './ibtmp1' size is now 12 MB.
InnoDB: 96 redo rollback segment(s) found. 1 redo rollback segment(s) are active.
InnoDB: 32 non-redo rollback segment(s) are active.
InnoDB: 5.7.26 started; log sequence number 496419657312

xtrabackup: starting shutdown with innodb_fast_shutdown = 1
InnoDB: FTS optimize thread exiting.
InnoDB: Starting shutdown...
InnoDB: Shutdown completed; log sequence number 496419682236
InnoDB: Number of pools: 1
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = .
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = .
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 524288000
InnoDB: PUNCH HOLE support available
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Uses event mutexes
InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
InnoDB: Compressed tables use zlib 1.2.3
InnoDB: Number of pools: 1
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M
InnoDB: Completed initialization of buffer pool
InnoDB: page_cleaner coordinator priority: -20
InnoDB: Setting log file ./ib_logfile101 size to 500 MB
InnoDB: Progress in MB:
100 200 300 400 500
InnoDB: Setting log file ./ib_logfile1 size to 500 MB
InnoDB: Progress in MB:
100 200 300 400 500
InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
InnoDB: New log files created, LSN=496419682236
InnoDB: Highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 496419682316
InnoDB: Doing recovery: scanned up to log sequence number 496419682325 (0%)
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
2019-11-14 10:24:04 0x7fab52862720 InnoDB: Assertion failure in thread 140373800658720 in file fsp0fsp.cc line 3864
InnoDB: Failing assertion: xdes_mtr_get_bit(descr, XDES_FREE_BIT, header_page % FSP_EXTENT_SIZE, mtr) == FALSE
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
02:24:04 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
/home/owsdb/xtrabackup/bin/xtrabackup(my_print_stacktrace+0x2c)[0xd9097c]
/home/owsdb/xtrabackup/bin/xtrabackup(handle_fatal_signal+0x262)[0xa46d12]
/lib64/libpthread.so.0[0x36aec0f7e0]
/lib64/libc.so.6(gsignal+0x35)[0x36ae8325e5]
/lib64/libc.so.6(abort+0x175)[0x36ae833dc5]
/home/owsdb/xtrabackup/bin/xtrabackup[0x713781]
/home/owsdb/xtrabackup/bin/xtrabackup(_Z14fseg_free_stepPhbP5mtr_t+0x5f5)[0xca5dc5]
/home/owsdb/xtrabackup/bin/xtrabackup[0xd5abbd]
/home/owsdb/xtrabackup/bin/xtrabackup(_Z8btr_freeRK9page_id_tRK11page_size_t+0x14d)[0xd5b14d]
/home/owsdb/xtrabackup/bin/xtrabackup(_Z27dict_drop_index_tree_in_memPK12dict_index_tm+0x5c)[0xcf08bc]
/home/owsdb/xtrabackup/bin/xtrabackup(_Z24row_drop_table_for_mysqlPKcP5trx_tbbP12dict_table_t+0x550)[0xb6e960]
/home/owsdb/xtrabackup/bin/xtrabackup(_Z26row_mysql_drop_temp_tablesv+0x32b)[0xb70b2b]
/home/owsdb/xtrabackup/bin/xtrabackup(_Z29recv_recovery_rollback_activev+0x2e)[0xbe371e]
/home/owsdb/xtrabackup/bin/xtrabackup(_Z34innobase_start_or_create_for_mysqlv+0x3d7a)[0xb31d5a]
/home/owsdb/xtrabackup/bin/xtrabackup[0x73a589]
/home/owsdb/xtrabackup/bin/xtrabackup[0x7473d9]
/home/owsdb/xtrabackup/bin/xtrabackup(main+0xe2c)[0x7210ec]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x36ae81ed1d]
/home/owsdb/xtrabackup/bin/xtrabackup[0x739025]

Please report a bug at https://jira.percona.com/projects/PXB

 

and then I use strace to start xtrabackup prepare, I see it crashed after read a tmp file:

open("./tmpdir/#sql331a_257dbad_52.ibd", O_RDONLY) = 11
lseek(11, 0, SEEK_CUR) = 0
lseek(11, 0, SEEK_END) = 98304
lseek(11, 0, SEEK_SET) = 0
pread(11, "?\247h7\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0t\5s\211\243\0\10\0\0\0\0\0\0"..., 16384, 0) = 16384
close(11) = 0
open("./tmpdir/#sql331a_257dbad_52.ibd", O_RDWR) = 11
pread(11, "\24vO^\0\0\0\3\377\377\377\377\377\377\377\377\0\0\0t\5s\211\243E\277\0\0\0\0\0\0"..., 16384, 49152) = 16384
pread(11, "\343s\222r\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0t\4\362W\261\0\5\0\0\0\0\0\0"..., 16384, 16384) = 16384
pread(11, "?\247h7\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0t\5s\211\243\0\10\0\0\0\0\0\0"..., 16384, 0) = 16384
open("/etc/localtime", O_RDONLY) = 13
fstat(13, {st_mode=S_IFREG|0644, st_size=388, ...}) = 0
fstat(13, {st_mode=S_IFREG|0644, st_size=388, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7efe5435b000
read(13, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\2\0\0\0\2\0\0\0\0"..., 4096) = 388
lseek(13, -240, SEEK_CUR) = 148
read(13, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\3\0\0\0\3\0\0\0\0"..., 4096) = 240
close(13) = 0
munmap(0x7efe5435b000, 4096) = 0
write(2, "2019-11-15 17:35:46 0x7efe541477"..., 342019-11-15 17:35:46 0x7efe54147720) = 34
write(2, " InnoDB: Assertion failure in t"..., 83 InnoDB: Assertion failure in thread 139630797420320 in file fsp0fsp.cc line 3864
) = 83
write(2, "InnoDB: Failing assertion: xdes_"..., 111InnoDB: Failing assertion: xdes_mtr_get_bit(descr, XDES_FREE_BIT, header_page % FSP_EXTENT_SIZE, mtr) == FALSE
) = 111
write(2, "InnoDB: We intentionally generat"..., 404InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
) = 404
rt_sigprocmask(SIG_UNBLOCK, [ABRT], NULL, 8) = 0
tgkill(25440, 25440, SIGABRT) = 0
— SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=25440, si_uid=0} —
write(2, "09:35:46 UTC - xtrabackup got si"..., 4109:35:46 UTC - xtrabackup got signal 6 ;
) = 41
write(2, "This could be because you hit a "..., 116This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
) = 116
write(2, "Attempting to collect some infor"..., 179Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

) = 179
write(2, "Thread pointer: 0x0\n", 20Thread pointer: 0x0
) = 20
write(2, "Attempting backtrace. You can us"..., 159Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
) = 159

 

then i stoped the mysql server,and move tmpdir outof datadir,then backup and prepare,it completed successfully.

 

 

Environment

None

Attachments

5

Smart Checklist

Activity

Show:

Julia Vural February 19, 2025 at 11:56 AM

It appears that this issue is no longer being worked on, so we are closing it for housekeeping purposes. If you believe the issue still exists, please open a new ticket after confirming it's present in the latest release.

Sergey Kuzmichev November 18, 2019 at 3:49 PM

Hello,

Thank you for your bug report.

Reproduced as described.

Tested with MySQL versions 5.6.23 and 5.6.46. MySQL 5.7 is not affected (due to shared temporary tablespace change).
xtrabackup version 2.4.16 based on MySQL server 5.7.26 Linux (x86_64) (revision id: c807cfa)

Steps to reproduce:
1. Run mysql 5.6 with innodb_file_per_table enabled.
2. Set up mysql to have tmpdir point to path inside datadir.
3. Create new session, create temporary table, leave session running.

4. Run xtrabackup --backup

5. Run xtrabackup --prepare, it will crash

Logs of backup and prepare from sandbox attached. Trace of prepare attached as well. Matches with the report.

Won't Do

Details

Assignee

Reporter

Affects versions

Priority

Smart Checklist

Created November 18, 2019 at 1:10 AM
Updated February 19, 2025 at 11:56 AM
Resolved February 19, 2025 at 11:56 AM