Rotating Innodb Master key post upgrade crashes the server

Description

Steps to repro:

1. create datadir on 5.7:

./bin/mysqld --no-defaults --datadir=`pwd`/data_57 --initialize-insecure --plugin-dir=/home/mohit.joshi/PS010819-percona-server-5.7.26-29-linux-x86_64-debug/lib/mysql/plugin/ --early-plugin-load=keyring_file.so --keyring_file_data=./data/mykey --innodb_sys_tablespace_encrypt=ON --innodb_undo_tablespaces=10

2. start the server on 5.7:

./bin/mysqld --defaults-file=~/my.cnf --datadir=`pwd`/data_57 --port=21000 --socket=/tmp/mysql_21000.sock --lc-messages-dir=./share --plugin-dir=/home/mohit.joshi/PS010819-percona-server-5.7.26-29-linux-x86_64-debug/lib/mysql/plugin/ --early-plugin-load=keyring_file.so --keyring_file_data=./data/mykey --max-connections=1024 --log-error --general-log --log-error-verbosity=3 --core-file --innodb_undo_log_encrypt=ON --innodb_temp_tablespace_encrypt=ON --innodb_redo_log_encrypt=MASTER_KEY --innodb_encrypt_tables=ON --innodb_sys_tablespace_encrypt=ON --innodb_undo_tablespaces=5 --innodb_parallel_dblwr_encrypt=ON --innodb_encrypt_online_alter_logs=ON --encrypt_tmp_files=ON &

3. Now run sysbench to run some Load:

mysql -A -uroot -S/tmp/mysql_21000.sock -e "create user sysbench@'%' identified with mysql_native_password by 'test';"
mysql -A -uroot -S/tmp/mysql_21000.sock -e "grant all on . to sysbench@'%';"
mysql -A -uroot -S/tmp/mysql_21000.sock -e "drop database if exists sbtest;create database sbtest;"

sysbench /usr/share/sysbench/oltp_insert.lua --mysql-db=sbtest --mysql-user=sysbench --mysql-password=test --db-driver=mysql --mysql-socket=/tmp/mysql_21000.sock --threads=1000 --tables=100 --table-size=1000 prepare

sysbench /usr/share/sysbench/oltp_read_write.lua --mysql-db=sbtest --mysql-user=sysbench --mysql-password=test --db-driver=mysql --mysql-socket=/tmp/mysql_21000.sock --threads=100 --tables=100 --time=3000 --report-interval=1 --events=1870000000 --db-ps-mode=disable run &

4. Shutdown the server from client

5. Start the server on 8.0 using 5.7 datadir:

./bin/mysqld --defaults-file=~/my.cnf --datadir=/home/mohit.joshi/PS200819-percona-server-5.7.27-30-linux-x86_64-debug/data_57 --port=22000 --socket=/tmp/mysql_22000.sock --lc-messages-dir=./share --plugin-dir=/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/lib/mysql/plugin/ --early-plugin-load=keyring_file.so --keyring_file_data=/home/mohit.joshi/PS200819-percona-server-5.7.27-30-linux-x86_64-debug/data_57/data/mykey --max-connections=1024 --log-error --general-log --log-error-verbosity=3 --core-file --innodb_undo_log_encrypt=ON --innodb_temp_tablespace_encrypt=ON --innodb_redo_log_encrypt=MASTER_KEY --default_table_encryption=ON --innodb_sys_tablespace_encrypt=ON --innodb_undo_tablespaces=2 --innodb_parallel_dblwr_encrypt=ON --innodb_encrypt_online_alter_logs=ON --encrypt_tmp_files=ON &

6. Execute on upgraded server:
ALTER INSTANCE ROTATE INNODB MASTER KEY;

 

Stack trace:

stack_bottom = 7f537c14dd18 thread_stack 0x46000
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(my_print_stacktrace(unsigned char*, unsigned long)+0x55) [0x4808cca]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(handle_fatal_signal+0x2c6) [0x363a5b3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f539bded390]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38) [0x7f539a0c5428]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f539a0c702a]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x1aa) [0x4db6309]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(buf_page_get_gen(page_id_t const&, page_size_t const&, unsigned long, buf_block_t*, Page_fetch, char const*, unsigned long, mtr_t*, bool, dberr_t*)+0x2d8) [0x4e650b8]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(fsp_header_write_encryption(unsigned int, unsigned long, unsigned char*, bool, bool, mtr_t*)+0x8e) [0x4fb6be6]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(fsp_header_rotate_encryption(fil_space_t*, unsigned char*, mtr_t*)+0xf5) [0x4fb6efb]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(encryption_rotate_low(fil_space_t*)+0xeb) [0x4f78cbe]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(Fil_system::encryption_rotate_in_a_shard(Fil_shard*)+0x248) [0x4f79070]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(Fil_system::encryption_rotate_all()+0x80) [0x4f79184]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(fil_encryption_rotate()+0x13) [0x4f791d5]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(innobase_encryption_key_rotation()+0x136) [0x4a5b842]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(Rotate_innodb_master_key::execute()+0x228) [0x335bd4c]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(Sql_cmd_alter_instance::execute(THD*)+0x172) [0x3350e04]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(mysql_execute_command(THD*, bool)+0x61fe) [0x34487ba]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(mysql_parse(THD*, Parser_state*, bool)+0x6cd) [0x344b28b]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x1571) [0x343f7b4]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(do_command(THD*)+0x50b) [0x343dbcd]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld() [0x3623dd1]
/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld() [0x4902b76]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f539bde36ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f539a19741d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f52c0097b38): alter instance rotate innodb master key

Environment

None

Attachments

1

Smart Checklist

Activity

Show:

Satya Bodapati August 23, 2019 at 2:01 PM

Done

Details

Assignee

Reporter

Time tracking

6h logged

Fix versions

Affects versions

Priority

Smart Checklist

Created August 22, 2019 at 11:37 AM
Updated March 6, 2024 at 11:53 AM
Resolved August 23, 2019 at 2:01 PM