Correct incremental prepare failure with logical redo by skipping the apply of logical redos (MLOG_TABLE_DYNAMIC_META) during the incremental prepare (except the last prepare).

Problem:
PXB process MLOG_TABLE_DYNAMIC_META log events as part of prepare. This
will make table->dirty_status = METADATA_DIRTY. As part of the shutdown
procedure, we will persist those changes to mysql.ibd. MySQL server that
originated the backup will continue processing data and merging them
into memory DDBuffer. Lather when a checkpoint happens on MySQL, it
persist those changes to mysql.ibd and generate proper redo logs for
those changes. However, since we have already persisted the changes we
had until that point the layout of our mysql.ibd might not match the
layout of the redo log we try to apply and a crash can happen.

Solution:
Skip applying logical redos (MLOG_TABLE_DYNAMIC_META) during the
incremental prepare (except the last prepare).

Use metadata_from_lsn for copying delta pages of mysql.ibd since we skip
applying logical records during prepare. These logical redos are
converted to regular redo and flushed to pages in mysql.ibd when the
server process a checkpoint. So we directly take the physical changes
made to innodb_dynamic_metadata since the last backup. Hence we copy
all changes to mysql.ibd since last backup start_lsn instead of last
backup end_lsn.

with the solution proposed, we take the physical changes (generated by MLOG_TABLE_DYNAMIC_META records). So we can safely ignore the logical redo records during pxb incremental prepare

Right, just to add what marce said, the core problem is the nature of MLOG_TABLE_DYNAMIC_META records. These are LOGICAL REDO records. ie. applying them will cause newer regular redo + changes to actual pages.  So these changes are applied post recovery.

Another problem  of these type of records is merging/grouping of operations in memory and then the conversion to on-disk.

The problem here is that we are processing MLOG_TABLE_DYNAMIC_META during prepare of incrementals. This is marking metadata of tables as dirty and later when we finish the incremental backup, as part of the shutdown, those records get persistent to disk / mysql.ibd. The problem happens later when the server process a checkpoint during the next incremental backup, it will generate redo logs to be applied directly to mysql.ibd and since we have written to mysql.ibd as part of the previous incremental backup shutdown, the layout of the records/pages might not match.

 and I have discussed a few ideas:

• Have a separate redo log to store buffered changes to DD. PXB will process this file as part of incremental backup - Not chosen because this might be a bit too complicated and overengineering 

• Always copy mysql.ibd as part of incremental backups (not only delta) - Not chosen because mysql.ibd store a lot of other information, like gtid_executed / list of tables ... on server with a lot of tables or too many gtid gaps, the size of the incremental can be big.

• Always copy delta changes from mysql.ibd only based on LSN from previous backup - We will give this option a try since this has most of the implementation already done. We need to adjust incremental filter to thread mysql.ibd separatelly. 

The main idea is:

Skip MLOG_TABLE_DYNAMIC_META events if --apply-log-only is set (this is incremental, but its not the last incremental)

Apply them only when this is either a full backup alone or the last incremental backup.

Looks like we are observing something..

1) Can you please confirm if PXB is writing to DDTabeBuffer as part of shutdown or did you mean only 'Server'? If PXB is doing the writes to DDTableBuffer, then we really have something.. (Also Check if PXB has log_checkpointer thread and that also starts writing to DDTableBuffer?) ?

2) Server can write to pages as part of shutdown. PXB will get this changes either via redo log or via 'delta pages'. Can you also please dump pages from inc delta files? See if we have copied a mysql.ibd page 5 as part of delta