Backups fail on Ubuntu Server 18.04 LTS after Upgrade from 2.4.24-1.bionic to 2.4.25-1.bionic

Description

I have a UrBackup Server 2.2.11 on FreeNAS.

I am taking a backup of a VM on ESXi of Ubuntu Server 18.04 LTS.

The VM has mySQL Server 5.7 installed.

Backups are nightly and have been working fine using Xtra Backup 2.4.

Yesterday, I ran a monthly update cycle that upgraded Xtra Backup 2.4.24-1.bionic to 2.4.25-1.bionic.

Since that upgrade, backups have continually failed.  After incremental backups started to fail, I tried full backups but they fail with the exact same details.

I ran a force install with apt to download back to the previous version and backups are working again.

Here is the forced downgrade command:

sudo apt install percona-xtrabackup-24=2.4.24-1.bionic -V

DB server uName:

Linux <hostname> 4.15.0-176-generic #185-Ubuntu SMP Tue Mar 29 17:40:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

 

A dpkg search for mysql resulted in this:

rc bacula-director-mysql 5.2.6+dfsg-9.1ubuntu3 amd64 network backup service - MySQL storage for Director
ii libdbd-mysql-perl 1:4.050-3.bionic amd64 Perl5 database interface to the MariaDB/MySQL database
rc libmysqlclient18:amd64 5.5.50-0ubuntu0.14.04.1 amd64 MySQL database client library
ii libmysqlclient20:amd64 5.7.37-0ubuntu0.18.04.1 amd64 MySQL database client library
rc libmysqlcppconn7 1.1.3-5 amd64 MySQL Connector for C++ (library)
ii mysql-client-5.7 5.7.37-0ubuntu0.18.04.1 amd64 MySQL database client binaries
ii mysql-client-core-5.7 5.7.37-0ubuntu0.18.04.1 amd64 MySQL database core client binaries
ii mysql-common 5.8+1.0.4 all MySQL database common files, e.g. /etc/mysql/my.cnf
ii mysql-server 5.7.37-0ubuntu0.18.04.1 all MySQL database server (metapackage depending on the latest version)
rc mysql-server-5.5 5.5.50-0ubuntu0.14.04.1 amd64 MySQL database server binaries and system database setup
ii mysql-server-5.7 5.7.37-0ubuntu0.18.04.1 amd64 MySQL database server binaries and system database setup
ii mysql-server-core-5.7 5.7.37-0ubuntu0.18.04.1 amd64 MySQL database server binaries
rc mysql-workbench 6.0.8+dfsg-2 amd64 MySQL Workbench - a visual database modeling, administration and queuing tool
ii mysqltuner 1.7.2-1 all high-performance MySQL tuning script

 

UrBackup provided this log:

2022-05-02 09:25:18(info): Starting unscheduled full file backup...

2022-05-02 09:26:18(info): Backing up "root" without snapshot.

2022-05-02 09:26:18(info): Indexing of "root" done. 37592 filesystem lookups 0 db lookups and 0 db updates

2022-05-02 09:26:18(info): <hostname>: Loading file list...

2022-05-02 09:26:19(info): <hostname>: Started loading files...

2022-05-02 09:26:20(info): Referencing snapshot on "<hostname>" for path "root" failed: FAILED

2022-05-02 09:32:35(warning): Error getting complete file "ozrOUZmYtjHCWHTMoZNG|root/usr/local/var/urbackup/data/filelist_new_0.ub" from <hostname>. Errorcode: CANNOT_OPEN_FILE (3)

2022-05-02 09:43:19(info): Waiting for file transfers...

2022-05-02 09:43:21(error): mariadbxtrabackup: Error reading tar header (1). Unexpected length 0

2022-05-02 09:43:20(error): mariadbxtrabackup: Starting binary backup of MariaDB at Mon May  2 09:43:19 EDT 2022...

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --tmpdir=/tmp

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup: recognized client arguments: --backup=1 --stream=tar --extra-lsndir=/var/lib/mariadbbackup/last.new --user=<redacted> --password=*

2022-05-02 09:43:21(error): mariadbxtrabackup: 220502 09:43:20  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup' as '<redacted>'  (using password: YES).

2022-05-02 09:43:21(error): mariadbxtrabackup: 220502 09:43:20  version_check Connected to MySQL server

2022-05-02 09:43:21(error): mariadbxtrabackup: 220502 09:43:20  version_check Executing a version check against the server...

2022-05-02 09:43:21(error): mariadbxtrabackup: 220502 09:43:20  version_check Done.

2022-05-02 09:43:21(error): mariadbxtrabackup: 220502 09:43:20 Connecting to MySQL server host: localhost, user: <redacted>, password: set, port: not set, socket: not set

2022-05-02 09:43:21(error): mariadbxtrabackup: Using server version 5.7.37-0ubuntu0.18.04.1

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup version 2.4.25 based on MySQL server 5.7.35 Linux (x86_64) (revision id: 90fe9d0)

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup: uses posix_fadvise().

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup: cd to /var/lib/mysql

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup: open files limit requested 0, set to 1024

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup: using the following InnoDB configuration:

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup:   innodb_data_home_dir = .

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup:   innodb_log_group_home_dir = ./

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup:   innodb_log_files_in_group = 2

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup:   innodb_log_file_size = 50331648

2022-05-02 09:43:21(error): mariadbxtrabackup: InnoDB: Number of pools: 1

2022-05-02 09:43:21(error): mariadbxtrabackup: 13:43:20 UTC - xtrabackup got signal 11 ;

2022-05-02 09:43:21(error): mariadbxtrabackup: This could be because you hit a bug or data is corrupted.

2022-05-02 09:43:21(error): mariadbxtrabackup: This error can also be caused by malfunctioning hardware.

2022-05-02 09:43:21(error): mariadbxtrabackup: Attempting to collect some information that could help diagnose the problem.

2022-05-02 09:43:21(error): mariadbxtrabackup: As this is a crash and something is definitely wrong, the information

2022-05-02 09:43:21(error): mariadbxtrabackup: collection process might fail.

2022-05-02 09:43:21(error): mariadbxtrabackup:

2022-05-02 09:43:21(error): mariadbxtrabackup: Thread pointer: 0x0

2022-05-02 09:43:21(error): mariadbxtrabackup: Attempting backtrace. You can use the following information to find out

2022-05-02 09:43:21(error): mariadbxtrabackup: where mysqld died. If you see no messages after this, something went

2022-05-02 09:43:21(error): mariadbxtrabackup: terribly wrong...

2022-05-02 09:43:21(error): mariadbxtrabackup: stack_bottom = 0 thread_stack 0x10000

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup(my_print_stacktrace+0x3b)[0x55e37367722b]

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup(handle_fatal_signal+0x285)[0x55e37332e735]

2022-05-02 09:43:21(error): mariadbxtrabackup: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f54aafac980]

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup(my_tmpdir+0xa)[0x55e37366cb5a]

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup(+0x5a66a5)[0x55e3730216a5]

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup(ds_open+0x12)[0x55e37301e0d2]

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup(_Z22xtrabackup_backup_funcv+0xf35)[0x55e373010545]

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup(main+0x1185)[0x55e372ff3ab5]

2022-05-02 09:43:21(error): mariadbxtrabackup: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f54a8db4c87]

2022-05-02 09:43:21(error): mariadbxtrabackup: xtrabackup(_start+0x2a)[0x55e3730074ea]

2022-05-02 09:43:21(error): mariadbxtrabackup:

2022-05-02 09:43:21(error): mariadbxtrabackup: Please report a bug at https://jira.percona.com/projects/PXB

2022-05-02 09:43:21(error): Script "mariadbxtrabackup" returned a nun-null value 2. Failing backup.

2022-05-02 09:43:21(error): Client <hostname> went offline.

2022-05-02 09:43:22(info): Waiting for file hashing and copying threads...

2022-05-02 09:43:24(info): Waiting for metadata download stream to finish

2022-05-02 09:43:24(error): Error saving metadata. Could not open output file at "/mnt/iS3-R10-UrBackup-Storage/<hostname>/220502-0925/.hashes/urbackup_backup_scripts/mariadbxtrabackup/.dir_metadata"

2022-05-02 09:43:24(info): Writing new file list...

2022-05-02 09:43:28(info): All metadata was present

2022-05-02 09:43:29(info): Transferred 5.57562 GB - Average speed: 46.8493 MBit/s

2022-05-02 09:43:29(info): Time taken for backing up client <hostname>: 18m 11s

2022-05-02 09:43:29(error): Backup failed

Environment

None

is duplicated by

Smart Checklist

Activity

Marcelo Altmann May 10, 2022 at 10:02 PM

Hi .  This is to inform you we have released 2.4.26 with this fix. Thanks for reporting it.

Marcelo Altmann May 2, 2022 at 5:52 PM

Hi ,

 

Thanks for your report. We are aware of this issue. We have a fix in place and are working on a new release of xtrabackup - Please check .

 

For now you can either:
1) Downgrade to 2.4.24
2) Use -s-tream=xbxtream instead of tar.
3) Do not use stream at all.

Duplicate

Details

Assignee

Reporter

Affects versions

Priority

Smart Checklist

Created May 2, 2022 at 4:52 PM
Updated March 6, 2024 at 6:26 PM
Resolved May 2, 2022 at 5:52 PM