SEGFAULT in 2.4.25
Description
Environment
The CI runs on ubuntu-18.04 x86_64. I was able to repeat the issue running the test locally as well:
$ cat /etc/*release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
Amazon Linux release 2 (Karoo)
$ uname -a
Linux ip-....compute.internal 4.14.256-197.484.amzn2.x86_64 #1 SMP Tue Nov 30 00:17:50 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ /usr/bin/xtrabackup --version
xtrabackup: recognized server arguments: --datadir=/var/lib/mysql
xtrabackup version 2.4.25 based on MySQL server 5.7.35 Linux (x86_64) (revision id: 90fe9d0)
$ rpm -q --whatprovides /usr/bin/xtrabackup
percona-xtrabackup-24-2.4.25-1.el7.x86_64
✔ ~/git/vitess [main|✔]
04:03 $ go test -v -failfast -timeout 10m vitess.io/vitess/go/test/endtoend/backup/xtrabackup
=== RUN TestXtrabackup
=== RUN TestXtrabackup/TestReplicaBackup
E0427 04:04:25.101424 25247 vtctlclient_process.go:186] Output:
E0427 04:04:25.100290 28425 main.go:94] E0427 04:04:25.100006 backup.go:107] E0427 04:04:25.097695 backup.go:128] backup is not usable, aborting it: [exit status 2
xtrabackup failed with error. Output=]
Backup Error: rpc error: code = Unknown desc = TabletManager.Backup on zone1-0000002093 error: xtrabackup failed with error. Output=: exit status 2: xtrabackup failed with error. Output=: exit status 2
E0427 04:04:25.100694 28425 main.go:103] remote error: rpc error: code = Unknown desc = TabletManager.Backup on zone1-0000002093 error: xtrabackup failed with error. Output=: exit status 2: xtrabackup failed with error. Output=: exit status 2
backup_utils.go:603:
Error Trace: backup_utils.go:603
backup_utils.go:222
Error: Expected nil, but got: &exec.ExitError{ProcessState:(*os.ProcessState)(0xc0004adc50), Stderr:[]uint8(nil)}
Test: TestXtrabackup/TestReplicaBackup
--- FAIL: TestXtrabackup (33.20s)
--- FAIL: TestXtrabackup/TestReplicaBackup (6.73s)
FAIL
FAIL vitess.io/vitess/go/test/endtoend/backup/xtrabackup 33.210s
FAIL
Smart Checklist
Activity
Matt Lord May 10, 2022 at 10:38 PM
Thanks again for the quick turnaround! I already have a PR open to move back to using latest: https://github.com/vitessio/vitess/pull/10241
Marcelo Altmann May 10, 2022 at 10:01 PM
Hi @Matt Lord . This is to inform you we have released 2.4.26 with this fix. Thanks for reporting it.
Matt Lord May 2, 2022 at 10:28 PM
FWIW, going the wget + dpkg route for now: https://github.com/vitessio/vitess/pull/10194
Matt Lord May 2, 2022 at 9:57 PM
The debs are there but the package metadata does not appear to be:
root@fce394c8c6bb:/# apt-cache madison percona-xtrabackup-24
percona-xtrabackup-24 | 2.4.25-1.bionic | http://repo.percona.com/percona/apt bionic/main amd64 Packages
percona-xtrabackup-24 | 2.4.25-1 | http://repo.percona.com/percona/apt bionic/main Sources
root@fce394c8c6bb:/# apt-get install percona-xtrabackup-24=2.4.24-1
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Version '2.4.24-1.bionic' for 'percona-xtrabackup-24' was not foundFor now I can go the ugly / brute force way since it's just temporary and we don't plan on changing the OS release used in that CI job within the next few weeks:
root@fce394c8c6bb:/# wget https://repo.percona.com/pxb-24/apt/pool/main/p/percona-xtrabackup-24/percona-xtrabackup-24_2.4.24-1.bionic_amd64.deb && dpkg -i percona-xtrabackup-24_2.4.24-1.bionic_amd64.debUnless you think the repo side of things is relatively easy to address? Please let me know if I can help in some way.
Matt Lord May 2, 2022 at 9:30 PM
Ah, I do see them here: https://repo.percona.com/pxb-24/apt/pool/main/p/percona-xtrabackup-24/
I'll look into explicitly trying to install 2.4.24 via the apt repo now then. Thanks again.
We use the latest version of xtrabackup 2.4 in the Vitess CI to test backups and restores: https://github.com/vitessio/vitess/blob/main/.github/workflows/cluster_endtoend_20.yml
Those tests started failing in the last 24 hours, which coincides with the brand new 2.4.25 release: https://docs.percona.com/percona-xtrabackup/2.4/release-notes/2.4/2.4.25.html
You can see the failed test runs here: https://github.com/vitessio/vitess/actions/workflows/cluster_endtoend_20.yml
And the Vitess issue here: https://github.com/vitessio/vitess/issues/10146
It appears to be caused by this change that is only in the latest 2.4 and 8.0 releases: https://github.com/percona/percona-xtrabackup/commit/eb296758c569f631c6853e04338d1781f46fa7a0
xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: recognized server arguments: --datadir=/vt/vtdataroot/vtroot_7301/vt_0000006411/data --innodb_data_home_dir=/vt/vtdataroot/vtroot_7301/vt_0000006411/innodb/data --innodb_log_group_home_dir=/vt/vtdataroot/vtroot_7301/vt_0000006411/innodb/logs --log_bin=/vt/vtdataroot/vtroot_7301/vt_0000006411/bin-logs/vt-0000006411-bin --server-id=1885372146 --tmpdir=/vt/vtdataroot/vtroot_7301/vt_0000006411/tmp xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: recognized client arguments: --backup=1 --socket=/vt/vtdataroot/vtroot_7301/vt_0000006411/mysql.sock --slave-info=1 --user=vt_dba --target-dir=/vt/vtdataroot/vtroot_7301/vt_0000006411/tmp --stream=tar --password=* xtrabackupengine.go:311] xtrabackup stderr: 220427 03:19:06 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/vt/vtdataroot/vtroot_7301/vt_0000006411/mysql.sock' as 'vt_dba' (using password: YES). xtrabackupengine.go:311] xtrabackup stderr: 220427 03:19:06 version_check Connected to MySQL server xtrabackupengine.go:311] xtrabackup stderr: 220427 03:19:06 version_check Executing a version check against the server... xtrabackupengine.go:311] xtrabackup stderr: 220427 03:19:06 version_check Done. xtrabackupengine.go:311] xtrabackup stderr: 220427 03:19:06 Connecting to MySQL server host: localhost, user: vt_dba, password: set, port: not set, socket: /vt/vtdataroot/vtroot_7301/vt_0000006411/mysql.sock xtrabackupengine.go:311] xtrabackup stderr: Using server version 5.7.37-log xtrabackupengine.go:311] xtrabackup stderr: xtrabackup version 2.4.25 based on MySQL server 5.7.35 Linux (x86_64) (revision id: 90fe9d0) xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: uses posix_fadvise(). xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: cd to /vt/vtdataroot/vtroot_7301/vt_0000006411/data xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: open files limit requested 0, set to 65535 xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: using the following InnoDB configuration: xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: innodb_data_home_dir = /vt/vtdataroot/vtroot_7301/vt_0000006411/innodb/data xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: innodb_log_group_home_dir = /vt/vtdataroot/vtroot_7301/vt_0000006411/innodb/logs xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: innodb_log_files_in_group = 2 xtrabackupengine.go:311] xtrabackup stderr: xtrabackup: innodb_log_file_size = 50331648 xtrabackupengine.go:311] xtrabackup stderr: InnoDB: Number of pools: 1 xtrabackupengine.go:311] xtrabackup stderr: 03:19:06 UTC - xtrabackup got signal 11 ; xtrabackupengine.go:311] xtrabackup stderr: This could be because you hit a bug or data is corrupted. xtrabackupengine.go:311] xtrabackup stderr: This error can also be caused by malfunctioning hardware. xtrabackupengine.go:311] xtrabackup stderr: Attempting to collect some information that could help diagnose the problem. xtrabackupengine.go:311] xtrabackup stderr: As this is a crash and something is definitely wrong, the information xtrabackupengine.go:311] xtrabackup stderr: collection process might fail. xtrabackupengine.go:311] xtrabackup stderr: xtrabackupengine.go:311] xtrabackup stderr: Thread pointer: 0x0 xtrabackupengine.go:311] xtrabackup stderr: Attempting backtrace. You can use the following information to find out xtrabackupengine.go:311] xtrabackup stderr: where mysqld died. If you see no messages after this, something went xtrabackupengine.go:311] xtrabackup stderr: terribly wrong... xtrabackupengine.go:311] xtrabackup stderr: stack_bottom = 0 thread_stack 0x10000 xtrabackupengine.go:311] xtrabackup stderr: xtrabackup(my_print_stacktrace+0x2c)[0xd730ec] xtrabackupengine.go:311] xtrabackup stderr: xtrabackup(handle_fatal_signal+0x286)[0xa37116] xtrabackupengine.go:311] xtrabackup stderr: /lib64/libpthread.so.0(+0x117e0)[0x7f46d12ef7e0] xtrabackupengine.go:311] xtrabackup stderr: xtrabackup(my_tmpdir+0x4e)[0xd69cae] xtrabackupengine.go:311] xtrabackup stderr: xtrabackup[0x738b94] xtrabackupengine.go:311] xtrabackup stderr: xtrabackup(ds_open+0x12)[0x735b42] xtrabackupengine.go:311] xtrabackup stderr: xtrabackup(_Z22xtrabackup_backup_funcv+0xda0)[0x72aa60] xtrabackupengine.go:311] xtrabackup stderr: xtrabackup(main+0xa94)[0x704c44] xtrabackupengine.go:311] xtrabackup stderr: /lib64/libc.so.6(__libc_start_main+0xea)[0x7f46cf0d713a] xtrabackupengine.go:311] xtrabackup stderr: xtrabackup[0x71ee04] xtrabackupengine.go:311] xtrabackup stderr: xtrabackupengine.go:311] xtrabackup stderr: Please report a bug at https://jira.percona.com/projects/PXBFor now we will have to try and pin the version in our CI to 2.24.24. But this is likely to affect others as well.
Please let me know if there's any other info we can provide, tests you'd like me to try, etc. Thanks!