Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

xtrabackup: Error writing file '<unopen fd>' (OS errno 32 - Broken pipe)

Description

Hi Team,

Percona Mysql server version 8.0.22-13
xtrabackup version 8.0.22-15

We are facing strange issue while performing incremental backups with xtrabackup.
1) We ae taking incremental backups every 15 minutes.
2) Issue is appearing randomly.
3) Xtrabackup logs attached.

Please help me to debug and solve this issues.

Thanks,
Jayesh

Environment

Production

Attachments

1
  • 20 Mar 2021, 07:02 AM

Smart Checklist

Activity

Show:

Lalit Choudhary August 5, 2021 at 1:21 PM

Cannot Reproduce

Lalit Choudhary June 2, 2021 at 1:56 PM
Edited

tried the various options and tests to interrupt  xtarbackup --stream=xbstream | xbcloud from s3 storage bucket side, still can't reproduce this.

 

other case:

I can see a similar error in the case when a backup directory with the same name exists with backup and retrying backup will fail with the below error,

xtrabackup: Redo Log Archiving is not set up. 210602 12:44:48 >> log scanned up to (382661384) xtrabackup: Generating a list of tablespaces xtrabackup: Generating a list of tablespaces Scanning './' Completed space ID check of 2 files. Allocated tablespace ID 1 for sys/sys_config, old maximum was 0 Using undo tablespace './undo_001'. Using undo tablespace './undo_002'. Opened 2 existing undo tablespaces. 210602 12:44:49 [01] Compressing and streaming ./ibdata1 210602 12:44:49 [01] ...done xtrabackup: Error writing file '<unopen fd>' (OS errno 32 - Broken pipe) [01] xtrabackup: Error: failed to copy datafile. 210602 12:44:49 >> log scanned up to (382661384)

 

Marcelo Altmann May 31, 2021 at 12:30 PM

 Looking into the logs of this issue, this hangs happens before the copy threads are started (which caused https://perconadev.atlassian.net/browse/PXB-2486#icft=PXB-2486). So it seems that they are different issues. 

Also, This one happens on 8.0, while https://perconadev.atlassian.net/browse/PXB-2486#icft=PXB-2486 cannot be reproducible on 8.0 at the moment.
Can you please check if this issue is still happening and do further troubleshooting?

Getting a perf / coredump while this hang is happening can help to understand where we are waiting on.

Lalit Choudhary April 28, 2021 at 2:52 PM

Hi Jayesh,

Thank you for the report.

Looking at the log there is a connection issue to a remote host,

210320 00:34:59 >> log scanned up to (68143075718) 210320 00:35:00 >> log scanned up to (68143075718) 210320 00:35:01 >> log scanned up to (68143075718) Connection to bkp1 closed by remote host. 210320 00:35:02 >> log scanned up to (68143075718) 210320 00:35:03 >> log scanned up to (68143075718)

Looks like PXB hang and finally failed. we already have bug report for the same issue.

https://jira.percona.com/browse/PXB-2486
https://jira.percona.com/browse/PXB-2404

Retrying backup again to make a successful backup.

Marking this issue a duplicate of https://perconadev.atlassian.net/browse/PXB-2486#icft=PXB-2486.
 

Cannot Reproduce

Details

Assignee

Reporter

Priority

Smart Checklist

Created March 20, 2021 at 7:07 AM
Updated March 6, 2024 at 6:43 PM
Resolved August 5, 2021 at 1:21 PM

Flag notifications