Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

xbcloud error Upload failed: backup is incomplete

Description

We are using 1.10 with cw enabled. Running multiple clusters in single namespace but also different namespaces. We have 3 pxc clusters in 1 namespace. We have restored one of them to test pitr . After PiTR restore backups started to fail on only that instance we did PiTR. What maybe the issue ?

2 other pxc clusters are in the same namespace and running without issues , recreating the pxc that fails against backup did not help. 

attaching the full backup log;

 

 

Environment

None

Attachments

7
  • 28 Mar 2022, 11:53 AM
  • 28 Mar 2022, 11:51 AM
  • 28 Mar 2022, 11:51 AM
  • 28 Mar 2022, 11:51 AM
  • 28 Mar 2022, 11:51 AM
  • 25 Mar 2022, 03:21 PM
  • 24 Mar 2022, 12:02 PM

Smart Checklist

Activity

Aleksej Trofimov April 8, 2022 at 7:54 PM

Rahul - you are awesome - yes we also came up to the same solution, however I would also propose to add debugging info into few places - like this check - e.g. basically answering "why" it is inconsistent and what went wrong =)

I suspect you will be doing a fix?

Rahul Malik April 8, 2022 at 9:26 AM

The fix is loop through all the files uploaded by xbcloud to ensure xtrabackup_tablespace
is uploaded.

Tomislav Plavcic April 8, 2022 at 7:14 AM

Hi !
Thank you for more info! Since I cannot reproduce this issue I have pinged developers of XtraBackup to see if they will come up with some fix proposal.

Aleksej Trofimov March 31, 2022 at 12:16 PM

Hey Tomislav,

After some debugging - here is the part of the log which is success: │

│ 2022-03-29 12:00:28.783 INFO: [SST script] 220329 12:00:27 xbcloud: successfully uploaded chunk: mysql-testb-testb/mysql-testb-2022-03-29-12:00:00-full/xtrabackup_info.lz4.00000000000000000001, size: 33 │ │ 2022-03-29 12:00:28.783 INFO: [SST script] 220329 12:00:27 xbcloud: successfully uploaded chunk: mysql-testb-testb/mysql-testb-2022-03-29-12:00:00-full/xtrabackup_tablespaces.lz4.00000000000000000000, size: 128 │ │ 2022-03-29 12:00:28.783 INFO: [SST script] 220329 12:00:27 xbcloud: successfully uploaded chunk: mysql-testb-testb/mysql-testb-2022-03-29-12:00:00-full/xtrabackup_tablespaces.lz4.00000000000000000001, size: 40

So the last chunk there is xtrabackup_tablespaces.lz4.00000000000000000001 - According to the https://github.com/percona/percona-xtrabackup/blob/percona-xtrabackup-8.0.25-17/storage/innobase/xtrabackup/src/xbcloud/xbcloud.cc#L743

It compares the last chunk to match backup_name + "/xtrabackup_tablespaces";

However this is the log for failing backup:

│ 2022-03-30 10:28:10.717 INFO: [SST script] 220330 10:28:09 xbcloud: successfully uploaded chunk: mysql-testa-testa/mysql-testa-2022-03-30-10:27:43-full/xtrabackup_tablespaces.lz4.00000000000000000001, size: 40 │ │ 2022-03-30 10:28:10.718 INFO: [SST script] 220330 10:28:09 xbcloud: successfully uploaded chunk: mysql-testa-testa/mysql-testa-2022-03-30-10:27:43-full/xtrabackup_tablespaces.lz4.00000000000000000000, size: 128 │ │ 2022-03-30 10:28:10.718 INFO: [SST script] 220330 10:28:09 xbcloud: successfully uploaded chunk: mysql-testa-testa/mysql-testa-2022-03-30-10:27:43-full/xtrabackup_logfile.lz4.00000000000000000000, size: 535

last chunk is "xtrabackup_logfile.lz4.00000000000000000000" which would not match on "xtrabackup_tablespaces" which would raise "Upload failed: backup is incomplete"

Cagri Biroglu March 28, 2022 at 11:53 AM

Done

Details

Assignee

Reporter

Time tracking

3h logged

Fix versions

Affects versions

Priority

Smart Checklist

Created March 24, 2022 at 12:03 PM
Updated March 6, 2024 at 6:27 PM
Resolved April 18, 2022 at 11:45 AM

Flag notifications