Xtrabackup streaming backup fails with log block numbers mismatch
Description
Environment
CentOS Linux release 8.2.2004 (Core)
Smart Checklist
Activity
Vairavaprakash S R August 5, 2021 at 4:58 PM
"it stops xtrabackup and streams the saved log file too" this is the part that's wrong in the documentation.
Also, are there any future plans to support multiple streams with xbstream?
Lalit Choudhary August 5, 2021 at 2:29 PMEdited
Hi
The following option in xtrabackup doc says, ref:https://www.percona.com/doc/percona-xtrabackup/8.0/xtrabackup_bin/xbk_option_reference.html#cmdoption-stream
--stream=FORMAT
Stream all backup files to the standard output in the specified format. Currently, this option only supports the — xbstream format.
whereas following doc says, xbstream to stream all of the data files(https://www.percona.com/doc/percona-xtrabackup/LATEST/xtrabackup_bin/backup.streaming.html)
xtrabackup uses xbstream to stream all of the data files to
STDOUT, in a specialxbstreamformat. After it finishes streaming all of the data files toSTDOUT, it stops xtrabackup and streams the saved log file too.
Will review it and update the doc with the required information if needed.
Vairavaprakash S R July 6, 2021 at 6:32 AM
Can we please update the documentation that data and logs are streamed at the same time?
Jira Bot July 1, 2021 at 3:57 PM
Hello ,
I'm jira-bot, Percona's automated helper script. Your bug report is important
to us but we've been unable to reproduce it, and asked you for more
information. If we haven't heard from you on this in 3 more weeks, the issue
will be automatically closed.
Vairavaprakash S R June 2, 2021 at 3:08 PM
We are transferring data through the network, so that might be the issue.
But one query that I have is it's mentioned in the docs that log streaming happens after data streaming is complete, but they seem to happen at the same time can you please confirm?
Only the log thread seems to be stuck, the data thread is still streaming any reasons why?
Details
Details
Assignee
Reporter
Vairavaprakash S RLabels
Components
Affects versions
Priority
MediumSmart Checklist
Open Smart Checklist
Smart Checklist
Hi Team,
We are trying to take backup of large MySQL instance using XtraBackup version 8.0.22-15 based on MySQL server 8.0.22 Linux (x86_64) (revision id: fea8a0e)
We are hitting into the below issue every time we try a streaming backup.
Command used - /usr/bin/xtrabackup --user=root -p --backup --datadir=<dataDir> --socket=<socketFile> --extra-lsndir="/tmp/lsnDir" --stream=xbstream
xtrabackup: error: log block numbers mismatch:
xtrabackup: error: expected log block no. 363355851, but got no. 413687487 from the log file.
xtrabackup: error: it looks like InnoDB log has wrapped around before xtrabackup could process all records due to either log copying being too slow, or log files being too small.
xtrabackup: Error: read_logfile() failed.
xtrabackup: error: log copyiing failed.
Seems like the log copying thread seems to be stuck and after few hours it failing with the above error.
Does XtraBackup stream data and log to STDOUT at the same time?
https://www.percona.com/doc/percona-xtrabackup/LATEST/xtrabackup_bin/backup.streaming.html#:~:text=Streaming%20mode%2C%20supported%20by%20Percona,for%20storage%20of%20the%20backup.
"xtrabackup uses xbstream to stream all of the data files to STDOUT, in a special xbstream format. After it finishes streaming all of the data files to STDOUT, it stops xtrabackup and streams the saved log file too."
The above documentation says that log streaming happens after data streaming, but we see that data and logs are written to STDOUT at the same time by different threads, can you please confirm this?
Regarding the log copy thread getting stuck, can you please help and let us know if this is a bug or whether we need to change something?