Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Under high write load, backup fails with "log block numbers mismatch" error

Description

Since the migration to PS 8.0.34 and the new redo log implementation, many servers with a high write load are now failing with the error "log block numbers mismatch".  The innodb_redo_log_capacity is set to 512GB and about 5MB/s is written to the redo.  With these numbers, we can rule out a rewind of the redo log.

Here's the output of the command:

You will notice a few extra lines, I added a few extra outputs to get insights to what is going on. If that can help, the block number mismatch is always 128.

The modifications are in the file redo_log.cc at around line 300:

The backup looked only at 8 of the 32 redo log files and then failed to find the start_lsn value.

Environment

None

AFFECTED CS IDs

CS0041833

Activity

Show:

Marcelo Altmann November 23, 2023 at 2:34 PM
Edited

How to reproduce wrap-around:

Adjust LSN to a multiple of 512G and remove a few bytes so you are close to the borderline.

 

 

blocks wrap after 1073741824

Marcelo Altmann November 23, 2023 at 12:33 PM

Hi  

Thanks for providing the requested information. This confirms you are seeing the issue we are fixing in this bug.

We are targeting a fix in the next release.

kwangsoo, kim November 21, 2023 at 3:39 AM

You have registered the information you requested.

1. 3.7T
2. innodb_redo_log_capacity = 75161927680
3, 4

Satya Bodapati November 20, 2023 at 11:57 AM

Can you please share the following details?

  1. Datadir size

  2. server redolog configuration

  3. Last LSN that is printed (log scanned up to)

  4.  block numbers that are mismatched.

Want to ensure that you are seeing the same issue or something different.

kwangsoo, kim November 20, 2023 at 11:20 AM

Thank you so much for your reply.
The Error-LOG you requested is not much different from the log I provided above.

If you want more detailed logs, let me know how to do it. 

Done

Details

Assignee

Reporter

Needs QA

Yes

Fix versions

Affects versions

Priority

Smart Checklist

Created October 12, 2023 at 8:42 PM
Updated November 18, 2024 at 10:54 AM
Resolved November 24, 2023 at 11:07 AM