Set crypt_data based on the destination table encryption status
General
Escalation
General
Escalation
Description
Environment
None
Attachments
5
loaddata_7338.inc
24 Sep, 2020
bug_7338-master.opt
24 Sep, 2020
bug_7338.test
24 Sep, 2020
metadata_7338.inc
24 Sep, 2020
runload_7338.inc
24 Sep, 2020
Activity
Show:
Satya Bodapati
September 25, 2020 at 1:49 PM
Breakpoint 1, main (argc=10, argv=0x7fffffffe2e8) at /home/satya.bodapati/PS-8.0-7143/sql/main.cc:25
25 int main(int argc, char **argv) { return mysqld_main(argc, argv); }
(gdb) b write_page0
Breakpoint 2 at 0x55555a67317c: file /home/satya.bodapati/PS-8.0-7143/storage/innobase/fil/fil0crypt.cc, line 874.
(gdb) c
Continuing.Thread 46 "mysqld" hit Breakpoint 2, fil_space_crypt_t::write_page0 (this=0x7fff3c049870, space=0x7fff3c049b20, page=0x7fffeca10000 "",
mtr=0x7fffe47b2d90, a_min_key_version=0, a_max_key_version=0, a_type=0) at /home/satya.bodapati/PS-8.0-7143/storage/innobase/fil/fil0crypt.cc:874
874 uint a_max_key_version, uint a_type) {
(gdb) p space->id
$1 = 5
(gdb) condition 2 space->id == 4
(gdb) c
(gdb) bt
#0 fil_space_crypt_t::write_page0 (this=0x7fff3cba1e50, space=0x7fff3c04fd20, page=0x7fffec9fc000 "�7�\212", mtr=0x7fffe47b2b70, a_min_key_version=0,
a_max_key_version=0, a_type=0) at /home/satya.bodapati/PS-8.0-7143/storage/innobase/fil/fil0crypt.cc:874
#1 0x000055555a674aee in fil_crypt_write_crypt_data_to_page0 (space=0x7fff3c04fd20)
at /home/satya.bodapati/PS-8.0-7143/storage/innobase/fil/fil0crypt.cc:1443
#2 0x000055555a674db1 in fil_crypt_exclude_tablespace_from_rotation_permanently (space=0x7fff3c04fd20)
at /home/satya.bodapati/PS-8.0-7143/storage/innobase/fil/fil0crypt.cc:1504
#3 0x000055555a0c88cb in innobase_alter_encrypt_tablespace (hton=0x55555f887eb0, thd=0x7fff3c001000, alter_info=0x7fffe47b3640,
old_dd_space=0x7fff3c04a208, new_dd_space=0x7fff3c18def8) at /home/satya.bodapati/PS-8.0-7143/storage/innobase/handler/ha_innodb.cc:16211
#4 0x000055555a0c928a in innodb_alter_tablespace (hton=0x55555f887eb0, thd=0x7fff3c001000, alter_info=0x7fffe47b3640, old_dd_space=0x7fff3c04a208,
new_dd_space=0x7fff3c18def8) at /home/satya.bodapati/PS-8.0-7143/storage/innobase/handler/ha_innodb.cc:16348
#5 0x000055555a0cb1e7 in innobase_alter_tablespace (hton=0x55555f887eb0, thd=0x7fff3c001000, alter_info=0x7fffe47b3640, old_ts_def=0x7fff3c04a208,
new_ts_def=0x7fff3c18def8) at /home/satya.bodapati/PS-8.0-7143/storage/innobase/handler/ha_innodb.cc:16874
#6 0x0000555558c52389 in Sql_cmd_alter_tablespace::execute (this=0x7fff3cb75248, thd=0x7fff3c001000)
at /home/satya.bodapati/PS-8.0-7143/sql/sql_tablespace.cc:998
#7 0x0000555558b44d56 in mysql_execute_command (thd=0x7fff3c001000, first_level=true) at /home/satya.bodapati/PS-8.0-7143/sql/sql_parse.cc:4790
#8 0x0000555558b479fd in mysql_parse (thd=0x7fff3c001000, parser_state=0x7fffe47b4bd0, update_userstat=false)
at /home/satya.bodapati/PS-8.0-7143/sql/sql_parse.cc:5617
#9 0x0000555558b3b9cf in dispatch_command (thd=0x7fff3c001000, com_data=0x7fffe47b5c80, command=COM_QUERY)
at /home/satya.bodapati/PS-8.0-7143/sql/sql_parse.cc:1860
#10 0x0000555558b39d8d in do_command (thd=0x7fff3c001000) at /home/satya.bodapati/PS-8.0-7143/sql/sql_parse.cc:1323
#11 0x0000555558d3f618 in handle_connection (arg=0x55555f91b270)
at /home/satya.bodapati/PS-8.0-7143/sql/conn_handler/connection_handler_per_thread.cc:308
#12 0x000055555a7c21ac in pfs_spawn_thread (arg=0x55555f9346e0) at /home/satya.bodapati/PS-8.0-7143/storage/perfschema/pfs.cc:2881
#13 0x00007ffff5de76db in start_thread (arg=0x7fffe47b6700) at pthread_create.c:463
#14 0x00007ffff5b10a3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) f 7
#7 0x0000555558b44d56 in mysql_execute_command (thd=0x7fff3c001000, first_level=true) at /home/satya.bodapati/PS-8.0-7143/sql/sql_parse.cc:4790
4790 res = lex->m_sql_cmd->execute(thd);
(gdb) p thd->query()
2020-09-25T13:43:16.141190Z 0 [Note] [MY-011953] [InnoDB] Page cleaner took 17736ms to flush 51 pages
$2 = (const LEX_CSTRING &) @0x7fff3c001218: {str = 0x7fff3cb740f8 "ALTER TABLESPACE tab02k_e ENCRYPTION 'N'", length = 40}Satya Bodapati
September 25, 2020 at 1:37 PM
(edited)
git blame on this code gives:
So the regression is from
commit 56fb352853e14cacb8952987dffefc819a9486d3
Author: Robert Golebiowski <robert.golebiowski@percona.com>
Date: Mon Jun 29 12:02:20 2020 +0200 PS-7140: crypt redo logs are not applied correctly
The redo log recovery works in two stages:
- parsing stage
- apply stage
During parsing stage we parse the log and add them into hash table. The
stact trace for parsing is:
-> recv_multi_rec
-> recv_parse_log_rec();
-> recv_parse_or_apply_log_rec_body(block == nullptr) -> thus no apply
-> recv_parse_log_rec();
-> recv_parse_or_apply_log_rec_body(block==nullptr) -> thus no apply
-> recv_add_to_hash_table
The recv_parse_or_apply_log_rec_body is called twice. Its task here is
to parse the logs, without applying them. This is because block passed
to this function is equal to nullptr. Each log record is added to
hash_table.https://github.com/percona/percona-server/commit/56fb352853e14cacb8952987dffefc819a9486d3
Satya Bodapati
September 25, 2020 at 1:35 PM
So the bug is in Fil_system::open_for_recovery() that thinks if crypt_data is found in hash, then tablespace is set as encrypted and of KEYRING type..
Since we have repeatable testcase, lets see who writes this crypt_data for space_id 4.
Satya Bodapati
September 25, 2020 at 1:29 PM
(edited)
a little further in that function we see
│995 uint type = mach_read_from_1(ptr); │
│996 ptr += 1; │
│997 │
>│998 ut_a(type == CRYPT_SCHEME_UNENCRYPTED || │
│999 type == CRYPT_SCHEME_1); // only supported(gdb) n
(gdb) p type
$29 = 0
So this means we do have crypt_data with type "CRYPT_SCHEME_UNENCRYPTED"
All encryption='n' tables & tablespaces (only in PS) have crypt_data written. So we are wrong to assume that just presence of crypt_data means the tablespace is encrypted and of type KEYRING.
While testing PS-8.0.21 release branch, I found a crash which looks like a regression. The crash is not seen prior to merge. Also, the assertion code does not exist on upstream.
The crash is seen with non-partition tables without the encryption=KEYRING feature.
Steps to reproduce (refer attached MTR testcase)
./pstress-ps --threads 1 --seconds 100 --tables 10 --records 100 --log-all-queries --log-failed-queries --no-partition-tables --rename-index 5 --rename-column 5 --rotate-master-key 50 --alt-tbs-enc 50 --only-cl-sqlRequest you to please take a look.
#0 __pthread_kill (threadid=<optimized out>, signo=6) at ../sysdeps/unix/sysv/linux/pthread_kill.c:62 #1 0x0000000004e83daa in my_write_core (sig=6) at /home/mohit.joshi/release-8.0.21/mysys/stacktrace.cc:409 #2 0x0000000003c3b387 in handle_fatal_signal (sig=6) at /home/mohit.joshi/release-8.0.21/sql/signal_handler.cc:194 #3 <signal handler called> #4 0x00007efc6d5c7438 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 #5 0x00007efc6d5c903a in __GI_abort () at abort.c:89 #6 0x0000000005373214 in ut_dbg_assertion_failed (expr=0x70c1178 "key_version_read_from_page != ENCRYPTION_KEY_VERSION_NOT_ENCRYPTED", file=0x70c0928 "/home/mohit.joshi/release-8.0.21/storage/innobase/os/os0file.cc", line=1767) at /home/mohit.joshi/release-8.0.21/storage/innobase/ut/ut0dbg.cc:98 #7 0x0000000005171f70 in load_key_needed_for_decryption (type=..., encryption=..., buf=0x7efc59eb4000 "\252\226\rX") at /home/mohit.joshi/release-8.0.21/storage/innobase/os/os0file.cc:1767 #8 0x0000000005172388 in os_file_io_complete (type=..., fh=22, buf=0x7efc59eb4000 "\252\226\rX", scratch=0x0, src_len=16384, offset=65536, len=16384) at /home/mohit.joshi/release-8.0.21/storage/innobase/os/os0file.cc:1842 #9 0x000000000517887c in os_file_io (in_type=..., file=22, buf=0x7efc59eb4000, n=16384, offset=65536, err=0x7efc6129ae94) at /home/mohit.joshi/release-8.0.21/storage/innobase/os/os0file.cc:5378 #10 0x00000000051792e9 in os_file_pread (type=..., file=22, buf=0x7efc59eb4000, n=16384, offset=65536, trx=0x0, err=0x7efc6129ae94) at /home/mohit.joshi/release-8.0.21/storage/innobase/os/os0file.cc:5542 #11 0x00000000051794d0 in os_file_read_page (type=..., file_name=0x7efc59ead640 "./tab16k_e.ibd", file=22, buf=0x7efc59eb4000, offset=65536, n=16384, o=0x0, exit_on_err=true, trx=0x0) at /home/mohit.joshi/release-8.0.21/storage/innobase/os/os0file.cc:5584 #12 0x000000000517a947 in os_file_read_func (type=..., file_name=0x7efc59ead640 "./tab16k_e.ibd", file=22, buf=0x7efc59eb4000, offset=65536, n=16384, trx=0x0) at /home/mohit.joshi/release-8.0.21/storage/innobase/os/os0file.cc:6035 #13 0x000000000517e12a in os_aio_func (type=..., aio_mode=AIO_mode::SYNC, name=0x7efc59ead640 "./tab16k_e.ibd", file=..., buf=0x7efc59eb4000, offset=65536, n=16384, read_only=false, m1=0x7efc59ead500, m2=0x0, space_id=5, trx=0x0, should_buffer=false) at /home/mohit.joshi/release-8.0.21/storage/innobase/os/os0file.cc:7574 #14 0x00000000055517d1 in pfs_os_aio_func (type=..., aio_mode=AIO_mode::SYNC, name=0x7efc59ead640 "./tab16k_e.ibd", file=..., buf=0x7efc59eb4000, offset=65536, n=16384, read_only=false, m1=0x7efc59ead500, m2=0x0, space_id=5, trx=0x0, should_buffer=false, src_file=0x71d1028 "/home/mohit.joshi/release-8.0.21/storage/innobase/fil/fil0fil.cc", src_line=8441) at /home/mohit.joshi/release-8.0.21/storage/innobase/include/os0file.ic:254 #15 0x00000000055663e9 in Fil_shard::do_io (this=0x7efc582e2b40, type=..., sync=true, page_id=..., page_size=..., byte_offset=0, len=16384, buf=0x7efc59eb4000, message=0x0, trx=0x0, should_buffer=false) at /home/mohit.joshi/release-8.0.21/storage/innobase/fil/fil0fil.cc:8438 #16 0x0000000005566a25 in _fil_io (type=..., sync=true, page_id=..., page_size=..., byte_offset=0, len=16384, buf=0x7efc59eb4000, message=0x0, trx=0x0, should_buffer=false) at /home/mohit.joshi/release-8.0.21/storage/innobase/fil/fil0fil.cc:8593 #17 0x0000000005445303 in dblwr_recover_page (dblwr_page_no=13, space=0x7efc59eadc20, page_no=4, page=0x7efc584ac000 "U\333", <incomplete sequence \356\275>) at /home/mohit.joshi/release-8.0.21/storage/innobase/buf/buf0dblwr.cc:1928 #18 0x00000000054459fb in dblwr::recv::Pages::recover (this=0x7efc5835bbf0, space=0x7efc59eadc20) at /home/mohit.joshi/release-8.0.21/storage/innobase/buf/buf0dblwr.cc:2045 #19 0x000000000544432e in dblwr::recv::recover (pages=0x7efc5835bbf0, space=0x7efc59eadc20)