Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Backup redo encrypt database heap-buffer-overflow

Description

Backup redo encryption enabled instance crash occasionally, Build the PXB 8.0 with asan then report heap-buffer-overflow in Encryption::encrypt_log_block:

the logs:

Warning: MySQL variable 'datadir' points to nonexistent directory '(null)'
Warning: option 'datadir' has different values:
'/data1/8306' in defaults file
'(null)' in SHOW VARIABLES
210329 17:56:37 Executing LOCK INSTANCE FOR BACKUP...
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /data1/8306
xtrabackup: open files limit requested 100000, set to 400000
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = /data1/8306/dbdata_raw/dbdata
xtrabackup: innodb_data_file_path = ibdata1:2G:autoextend
xtrabackup: innodb_log_group_home_dir = /data1/8306/logs/innodb
xtrabackup: innodb_log_files_in_group = 4
xtrabackup: innodb_log_file_size = 1073741824
xtrabackup: using O_DIRECT_NO_FSYNC
Number of pools: 1
210329 17:56:37 Added plugin 'keyring_file.so' to load list.
Read redo log encryption metadata successful.
210329 17:56:37 Connecting to MySQL server host: 9.30.17.130, user: backup, password: set, port: 8306, socket: /data1/8306/prod/mysql.sock
xtrabackup: Redo Log Archiving is not set up.
Starting to parse redo log at lsn = 701501508936
Recovery parsing buffer extended to 4194304.
Recovery parsing buffer extended to 8388608.
Recovery parsing buffer extended to 16777216.
Recovery parsing buffer extended to 33554432.
Recovery parsing buffer extended to 67108864.
Recovery parsing buffer extended to 134217728.
=================================================================
==70279==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f934ca2d9fc at pc 0x00000262d90a bp 0x7ffc99f154d0 sp 0x7ffc99f14c78
WRITE of size 4 at 0x7f934ca2d9fc thread T0
#0 0x262d909 in __interceptor_memcpy.part.234 (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x262d909)
#1 0x3a3c358 in Encryption::encrypt_log_block(IORequest const&, unsigned char*, unsigned char*) (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x3a3c358)
#2 0x3a3dca7 in Encryption::encrypt_log(IORequest const&, unsigned char*, unsigned long, unsigned char*, unsigned long*) (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x3a3dca7)
#3 0x278a8ed in Redo_Log_Writer::write_buffer(unsigned char*, unsigned long) (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x278a8ed)
#4 0x278ccac in Redo_Log_Data_Manager::start() (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x278ccac)
#5 0x2708100 in xtrabackup_backup_func() (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x2708100)
#6 0x25804b8 in main (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x25804b8)
#7 0x7f9358386c04 in __libc_start_main (/lib64/libc.so.6+0x21c04)
#8 0x25d5094 (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x25d5094)

0x7f934ca2d9fc is located 477 bytes to the right of 16842783-byte region [0x7f934ba1d800,0x7f934ca2d81f)
allocated by thread T0 here:
#0 0x2692ea8 in calloc (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x2692ea8)
#1 0x26e1b24 in ut_allocator<unsigned char>::allocate(unsigned long, unsigned char const*, unsigned int, bool, bool) (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x26e1b24)
#2 0x278d68c in aligned_memory<unsigned char, 65536ul>::allocate(unsigned long) (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x278d68c)
#3 0x278d91f in Redo_Log_Writer::Redo_Log_Writer() (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x278d91f)
#4 0x270798d in Redo_Log_Data_Manager::Redo_Log_Data_Manager() (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x270798d)
#5 0x2707f23 in xtrabackup_backup_func() (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x2707f23)
#6 0x25804b8 in main (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x25804b8)
#7 0x7f9358386c04 in __libc_start_main (/lib64/libc.so.6+0x21c04)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/data1/ragnar/XtraBackup/percona-xtrabackup/tdsql/mysql_install/mysql-server-8.0.18/bin/xtrabackup+0x262d909) in __interceptor_memcpy.part.234
Shadow bytes around the buggy address:
0x0ff2e993dae0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0ff2e993daf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0ff2e993db00: 00 00 00 07 fa fa fa fa fa fa fa fa fa fa fa fa
0x0ff2e993db10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0ff2e993db20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0ff2e993db30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa[fa]
0x0ff2e993db40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0ff2e993db50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0ff2e993db60: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0ff2e993db70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0ff2e993db80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==70279==ABORTING

Environment

None

Smart Checklist

Activity

Show:

Jira Bot June 20, 2021 at 12:56 PM

Hello ,
It's been 52 days since this issue went into Incomplete and we haven't heard
from you on this.

At this point, our policy is to Close this issue, to keep things from getting
too cluttered. If you have more information about this issue and wish to
reopen it, please reply with a comment containing "jira-bot=reopen".

Jira Bot June 12, 2021 at 12:56 PM

Hello ,
It's jira-bot again. Your bug report is important to us, but we haven't heard
from you since the previous notification. If we don't hear from you on
this in 7 days, the issue will be automatically closed.

Jira Bot May 28, 2021 at 11:56 AM

Hello ,
I'm jira-bot, Percona's automated helper script. Your bug report is important
to us but we've been unable to reproduce it, and asked you for more
information. If we haven't heard from you on this in 3 more weeks, the issue
will be automatically closed.

Lalit Choudhary April 29, 2021 at 10:59 AM

Hi

Thank you for the report.

Please send reproducible steps with following information,

PXB version, myql version,  mysql config, xtrabackup command that you are using.

 

Also check for this warning, make sure xtrabackup reading from correct data-dir

kong zhi March 29, 2021 at 11:12 AM

Scratch_buf of Redo_log_writer is initialized 16M, but log_buf store the redo log before encryption is redo_log_read_buffer_size which can be configured.

Incomplete

Details

Assignee

Reporter

Priority

Smart Checklist

Created March 29, 2021 at 10:20 AM
Updated March 6, 2024 at 6:43 PM
Resolved June 20, 2021 at 12:56 PM