All work

Select view

Select search mode

 
50 of

Incremental backup prepare crash when a partitioned table column is compressed

Won't Do

Description

MS8.0.21 is running as

/home/mchawla/MS_8_0_21/bin/mysqld --no-defaults --core-file --basedir=/home/mchawla/MS_8_0_21 --tmpdir=/home/mchawla/MS_8_0_21/data --datadir=/home/mchawla/MS_8_0_21/data --socket=/home/mchawla/MS_8_0_21/socket.sock --port=13966 --log-error=/home/mchawla/MS_8_0_21/log/master.err --server-id=100 --log-bin=binlog

Create 10 sysbench tables

Create 3 partitioned tables and change the partitions

/home/mchawla/MS_8_0_21/bin/mysql -uroot -S/home/mchawla/MS_8_0_21/socket.sock -e "DROP TABLE IF EXISTS sbtest1; DROP TABLE IF EXISTS sbtest2; DROP TABLE IF EXISTS sbtest3;" test /home/mchawla/MS_8_0_21/bin/mysql -uroot -S/home/mchawla/MS_8_0_21/socket.sock -e "CREATE TABLE sbtest1 (id int NOT NULL AUTO_INCREMENT, k int NOT NULL DEFAULT '0', c char(120) NOT NULL DEFAULT '', pad char(60) NOT NULL DEFAULT '', PRIMARY KEY (id), KEY k_1 (k) ) PARTITION BY HASH(id) PARTITIONS 10;" test /home/mchawla/MS_8_0_21/bin/mysql -uroot -S/home/mchawla/MS_8_0_21/socket.sock -e "CREATE TABLE sbtest2 (id int NOT NULL AUTO_INCREMENT, k int NOT NULL DEFAULT '0', c char(120) NOT NULL DEFAULT '', pad char(60) NOT NULL DEFAULT '', PRIMARY KEY (id), KEY k_1 (k) ) PARTITION BY RANGE(id) (PARTITION p0 VALUES LESS THAN (500), PARTITION p1 VALUES LESS THAN (1000), PARTITION p2 VALUES LESS THAN MAXVALUE);" test /home/mchawla/MS_8_0_21/bin/mysql -uroot -S/home/mchawla/MS_8_0_21/socket.sock -e "CREATE TABLE sbtest3 (id int NOT NULL AUTO_INCREMENT, k int NOT NULL DEFAULT '0', c char(120) NOT NULL DEFAULT '', pad char(60) NOT NULL DEFAULT '', PRIMARY KEY (id), KEY k_1 (k) ) PARTITION BY KEY() PARTITIONS 5;" test sysbench /usr/share/sysbench/oltp_insert.lua --tables=3 --mysql-db=test --mysql-user=root --threads=100 --db-driver=mysql --mysql-socket=/home/mchawla/MS_8_0_21/socket.sock --time=5 run >/dev/null 2>&1 ( for ((i=1; i<=10; i++)); do /home/mchawla/MS_8_0_21/bin/mysql -uroot -S/home/mchawla/MS_8_0_21/socket.sock -e "ALTER TABLE test.sbtest1 COALESCE PARTITION 5;" >/dev/null 2>&1 /home/mchawla/MS_8_0_21/bin/mysql -uroot -S/home/mchawla/MS_8_0_21/socket.sock -e "ALTER TABLE test.sbtest1 PARTITION BY HASH(id) PARTITIONS 10;" >/dev/null 2>&1 done ) & ( for ((i=1; i<=10; i++)); do /home/mchawla/MS_8_0_21/bin/mysql -uroot -S/home/mchawla/MS_8_0_21/socket.sock -e "ALTER TABLE test.sbtest2 DROP PARTITION p2;" >/dev/null 2>&1 /home/mchawla/MS_8_0_21/bin/mysql -uroot -S/home/mchawla/MS_8_0_21/socket.sock -e "ALTER TABLE test.sbtest2 ADD PARTITION (PARTITION p2 VALUES LESS THAN MAXVALUE);" >/dev/null 2>&1 done ) & ( for ((i=1; i<=10; i++)); do /home/mchawla/MS_8_0_21/bin/mysql -uroot -S/home/mchawla/MS_8_0_21/socket.sock -e "ALTER TABLE test.sbtest3 REBUILD PARTITION p0, p1;" >/dev/null 2>&1 /home/mchawla/MS_8_0_21/bin/mysql -uroot -S/home/mchawla/MS_8_0_21/socket.sock -e "ALTER TABLE test.sbtest3 OPTIMIZE PARTITION p2;" >/dev/null 2>&1 /home/mchawla/MS_8_0_21/bin/mysql -uroot -S/home/mchawla/MS_8_0_21/socket.sock -e "ALTER TABLE test.sbtest3 ANALYZE PARTITION p3,p4;" >/dev/null 2>&1 done ) &

Take full backup with lock-ddl

/home/mchawla/pxb_8_0_14_debug/bin/xtrabackup --user=root --password= --backup --target-dir=/home/mchawla/dbbackup_01_09_2020/full -S /home/mchawla/MS_8_0_21/socket.sock --datadir=/home/mchawla/MS_8_0_21/data --lock-ddl 2>/home/mchawla/backuplogs/full_backup_01_09_2020_16_log

Run load

sysbench /usr/share/sysbench/oltp_insert.lua --tables=10 --mysql-db=test --mysql-user=root --threads=50 --db-driver=mysql --mysql-socket=/home/mchawla/MS_8_0_21/socket.sock --time=20 --rand-type=uniform run >/dev/null 2>&1 & sleep 10

Take incremental backup with lock-ddl

/home/mchawla/pxb_8_0_14_debug/bin/xtrabackup --user=root --password= --backup --target-dir=/home/mchawla/dbbackup_01_09_2020/inc --incremental-basedir=/home/mchawla/dbbackup_01_09_2020/full -S /home/mchawla/MS_8_0_21/socket.sock --datadir=/home/mchawla/MS_8_0_21/data --lock-ddl 2>/home/mchawla/backuplogs/inc_backup_01_09_2020_16_log

Prepare full backup

/home/mchawla/pxb_8_0_14_debug/bin/xtrabackup --user=root --password= --prepare --apply-log-only --target_dir=/home/mchawla/dbbackup_01_09_2020/full 2>/home/mchawla/backuplogs/prepare_full_backup_01_09_2020_16_log

Prepare incremental backup

/home/mchawla/pxb_8_0_14_debug/bin/xtrabackup --user=root --password= --prepare --target_dir=/home/mchawla/dbbackup_01_09_2020/full --incremental-dir=/home/mchawla/dbbackup_01_09_2020/inc 2>/home/mchawla/backuplogs/prepare_inc_backup_01_09_2020_16_log

Stop mysql server and move the data directory

Restore full backup

/home/mchawla/pxb_8_0_14_debug/bin/xtrabackup --user=root --password= --copy-back --target-dir=/home/mchawla/dbbackup_01_09_2020/full --datadir=/home/mchawla/MS_8_0_21/data 2>/home/mchawla/backuplogs/restore_backup_01_09_2020_16_log

Start mysql server

Apply binlogs from xtrabackup position
Compress a partitioned table column

( for ((i=1; i<=10; i++)); do /home/mchawla/MS_8_0_21/bin/mysql -uroot -S/home/mchawla/MS_8_0_21/socket.sock -e "ALTER TABLE test.sbtest1 MODIFY c VARCHAR(250) COLUMN_FORMAT COMPRESSED NOT NULL DEFAULT '';" >/dev/null 2>&1 /home/mchawla/MS_8_0_21/bin/mysql -uroot -S/home/mchawla/MS_8_0_21/socket.sock -e "ALTER TABLE test.sbtest1 MODIFY c CHAR(120) COLUMN_FORMAT DEFAULT NOT NULL DEFAULT '';" >/dev/null 2>&1 done ) &

Take full backup

/home/mchawla/pxb_8_0_14_debug/bin/xtrabackup --user=root --password= --backup --target-dir=/home/mchawla/dbbackup_01_09_2020/full -S /home/mchawla/MS_8_0_21/socket.sock --datadir=/home/mchawla/MS_8_0_21/data 2>/home/mchawla/backuplogs/full_backup_01_09_2020_20_log

Run load

sysbench /usr/share/sysbench/oltp_insert.lua --tables=10 --mysql-db=test --mysql-user=root --threads=50 --db-driver=mysql --mysql-socket=/home/mchawla/MS_8_0_21/socket.sock --time=20 --rand-type=uniform run >/dev/null 2>&1 & sleep 10

Take incremental backup

/home/mchawla/pxb_8_0_14_debug/bin/xtrabackup --user=root --password= --backup --target-dir=/home/mchawla/dbbackup_01_09_2020/inc --incremental-basedir=/home/mchawla/dbbackup_01_09_2020/full -S /home/mchawla/MS_8_0_21/socket.sock --datadir=/home/mchawla/MS_8_0_21/data 2>/home/mchawla/backuplogs/inc_backup_01_09_2020_20_log

Prepare full backup

/home/mchawla/pxb_8_0_14_debug/bin/xtrabackup --user=root --password= --prepare --apply-log-only --target_dir=/home/mchawla/dbbackup_01_09_2020/full 2>/home/mchawla/backuplogs/prepare_full_backup_01_09_2020_20_log

Prepare incremental backup

/home/mchawla/pxb_8_0_14_debug/bin/xtrabackup --user=root --password= --prepare --target_dir=/home/mchawla/dbbackup_01_09_2020/full --incremental-dir=/home/mchawla/dbbackup_01_09_2020/inc 2>/home/mchawla/backuplogs/prepare_inc_backup_01_09_2020_20_log

PXB crashes

InnoDB: Assertion failure: page0page.ic:593:page_offset(rec) <= page_header_get_field(page, PAGE_HEAP_TOP) InnoDB: thread 140684854028032InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to https://jira.percona.com/projects/PXB. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 14:05:26 UTC - mysqld got signal 6 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x46000 /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(my_print_stacktrace(unsigned char const*, unsigned long)+0x43) [0x4bec7ff] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(handle_fatal_signal+0x299) [0x356cd79] /usr/lib64/libpthread.so.0(+0xf630) [0x7ff3c95b3630] /usr/lib64/libc.so.6(gsignal+0x37) [0x7ff3c722b387] /usr/lib64/libc.so.6(abort+0x148) [0x7ff3c722ca78] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup() [0x3c074c0] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup() [0x3a58184] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup() [0x3a57dd1] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(page_cur_insert_rec_low(unsigned char*, dict_index_t*, unsigned char const*, unsigned long*, mtr_t*)+0x22c) [0x3a5bfaa] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup() [0x3a59329] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(page_cur_parse_insert_rec(unsigned long, unsigned char const*, unsigned char const*, buf_block_t*, dict_index_t*, mtr_t*)+0x88b) [0x3a5bcce] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup() [0x3a0e35f] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(recv_recover_page_func(bool, buf_block_t*)+0x7e0) [0x3a102a6] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(buf_page_io_complete(buf_page_t*, bool)+0x663) [0x3713910] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(fil_aio_wait(unsigned long)+0x1f3) [0x3830d66] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup() [0x3b71865] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(void std::__invoke_impl<void, void (*&)(unsigned long), unsigned long&>(std::__invoke_other, void (*&)(unsigned long), unsigned long&)+0x34) [0x2df0b9e] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(std::__invoke_result<void (*&)(unsigned long), unsigned long&>::type std::__invoke<void (*&)(unsigned long), unsigned long&>(void (*&)(unsigned long), unsigned long&)+0x3f) [0x2df05d7] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(void std::_Bind<void (*(unsigned long))(unsigned long)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>)+0x48) [0x2defed4] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(void std::_Bind<void (*(unsigned long))(unsigned long)>::operator()<, void>()+0x2a) [0x2def23e] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(void Runnable::operator()<void (*)(unsigned long), unsigned long>(void (*&&)(unsigned long), unsigned long&&)+0xcb) [0x2ded98f] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(void std::__invoke_impl<void, Runnable, void (*)(unsigned long), unsigned long>(std::__invoke_other, Runnable&&, void (*&&)(unsigned long), unsigned long&&)+0x4f) [0x2deb0d7] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(std::__invoke_result<Runnable, void (*)(unsigned long), unsigned long>::type std::__invoke<Runnable, void (*)(unsigned long), unsigned long>(Runnable&&, void (*&&)(unsigned long), unsigned long&&)+0x57) [0x2de7b53] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(decltype (__invoke((_S_declval<0ul>)(), (_S_declval<1ul>)(), (_S_declval<2ul>)())) std::thread::_Invoker<std::tuple<Runnable, void (*)(unsigned long), unsigned long> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>)+0x5f) [0x2df1b5b] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(std::thread::_Invoker<std::tuple<Runnable, void (*)(unsigned long), unsigned long> >::operator()()+0x1d) [0x2df18b9] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup(std::thread::_State_impl<std::thread::_Invoker<std::tuple<Runnable, void (*)(unsigned long), unsigned long> > >::_M_run()+0x1c) [0x2df152a] /home/mchawla/pxb_8_0_14_debug/bin/xtrabackup() [0x4f8a1df] /usr/lib64/libpthread.so.0(+0x7ea5) [0x7ff3c95abea5] /usr/lib64/libc.so.6(clone+0x6d) [0x7ff3c72f38dd]

Logs are attached.

Note: Issue observed in the previous version also.

Environment

None

Attachments

3
  • 03 Dec 2020, 05:43 AM
  • 02 Sep 2020, 09:37 AM
  • 01 Sep 2020, 04:14 PM

Details

Assignee

Reporter

Time tracking

1w 2d 3h 15m logged

Affects versions

Priority

Created September 1, 2020 at 4:16 PM
Updated March 6, 2024 at 6:53 PM
Resolved January 11, 2021 at 6:55 PM

Activity

Marcelo Altmann 
January 11, 2021 at 6:55 PM

This issue doesn't happens with --lock-ddl enabled. We won't fix it for now

Manish Chawla 
December 3, 2020 at 5:49 AM

Reproduced the same crash with the latest PXB8.0.22 and MS8.0.22 running with keyring_file encryption.
Initialize database with encryption

/home/mchawla/MS_8_0_22/bin/mysqld --no-defaults --core-file --basedir=/home/mchawla/MS_8_0_22 --tmpdir=/home/mchawla/MS_8_0_22/data --datadir=/home/mchawla/MS_8_0_22/data --socket=/home/mchawla/MS_8_0_22/socket.sock --port=16922 --log-error=/home/mchawla/MS_8_0_22/log/master.err --server-id=100 --early-plugin-load=keyring_file.so --keyring_file_data=/home/mchawla/MS_8_0_22/keyring --innodb-undo-log-encrypt --innodb-redo-log-encrypt --default-table-encryption=ON --log-slave-updates --gtid-mode=ON --enforce-gtid-consistency --binlog-format=row --master_verify_checksum=ON --binlog_checksum=CRC32 --binlog-rotate-encryption-master-key-at-startup --table-encryption-privilege-check=ON --innodb-extend-and-initialize=OFF

Run pstress

./pstress-ps --tables 10 --logdir="${logdir}" --records 200 --threads 10 --seconds 50 --socket "${mysqldir}"/socket.sock --undo-tbs-sql 0

Take full backup and 6 incremental backups with lock-ddl

Prepare the full backup and incremental backups. PXB crashes during prepare of the fourth incremental backup

Read encryption metadata from ./test/tt_2_p#p#p1.ibd successfully, encryption of this tablespace enabled. InnoDB: Assertion failure: page0page.ic:593:page_offset(rec) <= page_header_get_field(page, PAGE_HEAP_TOP) InnoDB: thread 139819283900160InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to https://jira.percona.com/projects/PXB. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 04:58:48 UTC - mysqld got signal 6 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x46000 /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(my_print_stacktrace(unsigned char const*, unsigned long)+0x43) [0x4c4fc87] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(handle_fatal_signal+0x299) [0x359ec65] /usr/lib64/libpthread.so.0(+0xf630) [0x7f2a424ec630] /usr/lib64/libc.so.6(gsignal+0x37) [0x7f2a4015f387] /usr/lib64/libc.so.6(abort+0x148) [0x7f2a40160a78] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup() [0x3c4c66d] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup() [0x3a9d0ee] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup() [0x3a9cd94] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup() [0x3a9d506] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(page_cur_insert_rec_low(unsigned char*, dict_index_t*, unsigned char const*, unsigned long*, mtr_t*)+0x5af) [0x3aa129a] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup() [0x3a9e293] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(page_cur_parse_insert_rec(unsigned long, unsigned char const*, unsigned char const*, buf_block_t*, dict_index_t*, mtr_t*)+0x88e) [0x3aa0c3b] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup() [0x3a51a6a] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(recv_recover_page_func(bool, buf_block_t*)+0x7e3) [0x3a539db] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(buf_page_io_complete(buf_page_t*, bool)+0x663) [0x374e1f8] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(fil_aio_wait(unsigned long)+0x1f3) [0x386d12b] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup() [0x3bb606e] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(void std::__invoke_impl<void, void (*&)(unsigned long), unsigned long&>(std::__invoke_other, void (*&)(unsigned long), unsigned long&)+0x34) [0x2e09fec] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(std::__invoke_result<void (*&)(unsigned long), unsigned long&>::type std::__invoke<void (*&)(unsigned long), unsigned long&>(void (*&)(unsigned long), unsigned long&)+0x3f) [0x2e09a25] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(void std::_Bind<void (*(unsigned long))(unsigned long)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>)+0x48) [0x2e09322] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(void std::_Bind<void (*(unsigned long))(unsigned long)>::operator()<, void>()+0x2a) [0x2e0868c] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(void Runnable::operator()<void (*)(unsigned long), unsigned long>(void (*&&)(unsigned long), unsigned long&&)+0xcb) [0x2e06dad] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(void std::__invoke_impl<void, Runnable, void (*)(unsigned long), unsigned long>(std::__invoke_other, Runnable&&, void (*&&)(unsigned long), unsigned long&&)+0x4f) [0x2e04491] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(std::__invoke_result<Runnable, void (*)(unsigned long), unsigned long>::type std::__invoke<Runnable, void (*)(unsigned long), unsigned long>(Runnable&&, void (*&&)(unsigned long), unsigned long&&)+0x57) [0x2e00e49] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(decltype (__invoke((_S_declval<0ul>)(), (_S_declval<1ul>)(), (_S_declval<2ul>)())) std::thread::_Invoker<std::tuple<Runnable, void (*)(unsigned long), unsigned long> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>)+0x5f) [0x2e0afa9] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(std::thread::_Invoker<std::tuple<Runnable, void (*)(unsigned long), unsigned long> >::operator()()+0x1d) [0x2e0ad07] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup(std::thread::_State_impl<std::thread::_Invoker<std::tuple<Runnable, void (*)(unsigned long), unsigned long> > >::_M_run()+0x1c) [0x2e0a978] /home/mchawla/pxb_8_0_22_debug/bin/xtrabackup() [0x5003cff] /usr/lib64/libpthread.so.0(+0x7ea5) [0x7f2a424e4ea5] /usr/lib64/libc.so.6(clone+0x6d) [0x7f2a4022796d]

Note: Lock-ddl did not prevent the issue from occurring.

Logs attached as pxb2267_pstress.tar.gz

Manish Chawla 
November 27, 2020 at 12:27 PM

 I have copied the script and logs on the server at /home/pxb2267.

Marcelo Altmann 
November 26, 2020 at 4:15 PM

 can you prepare the script to reproduce it on qa2?

Manish Chawla 
November 25, 2020 at 2:33 PM

 I had run this with MS8.0.22, but the issue occurred again.

Flag notifications