Parallelize incremental delta apply in incremental backup prepare

Dear Percona Support, I'm experiencing significant performance delays when applying incremental backups using Percona XtraBackup on Ubuntu 24.04. The current process, which involves applying multiple incremental backups sequentially, takes approximately 237 minutes to complete. Observations: System Details: Ubuntu 24.04 MySQL 8.4.3 Database: 453 Schemas 158,771 Tables 1.5 TB Total Size Backup Strategy: xtrabackup with following options: --parallel=48 --use-memory=100G --prepare (for both full and incremental backups) --apply-log-only Specific target directories for full and incremental backups Performance Bottleneck: The primary bottleneck appears to be in the Applying stage of incremental backups, specifically when processing delta files. Proposed Improvement: Is it possible to explore parallelizing the prepare stage, particularly the steps that involve applying and deleting deltas? This could potentially involve: Applying deltas in parallel for different tables or files. Handling non-parallelizable tasks (e.g., updating metadata) sequentially after delta application is complete. Potential Benefits: We anticipate that such an optimization could significantly reduce the overall incremental backup time. This would lead to: Improved operational efficiency Reduced backup windows Enhanced database availability during backups Further Information: Here's an example log snippet illustrating the current Applying stage: 2024-12-18T15:44:50.819185-03:00 0 [Note] [MY-011825] [Xtrabackup] page size for /backup-mysql/bkp-xtrabackup/incr/2024-12-14_08-05-01/2024-12-14_16-05-02/com/transaction#p#2024_06.ibd.delta is 16384 bytes 2024-12-18T15:44:50.819312-03:00 0 [Note] [MY-011825] [Xtrabackup] Applying /backup-mysql/bkp-xtrabackup/incr/2024-12-14_08-05-01/2024-12-14_16-05-02/com/transaction#p#2024_06.ibd.delta to ./com/transaction#p#2024_06.ibd We appreciate your time and support in exploring this optimization opportunity. Please let us know if you require further information or if there are alternative approaches to consider.

Performance: Reduced Prepare Time: The total time required to apply all incremental backups (using the --prepare option) must be reduced by at least 40% compared to the current implementation, under similar system load and data volume conditions. This will be measured by comparing the execution time of the xtrabackup command with and without the optimization applied, using the provided test environment and dataset. Scalability: The optimization should demonstrate consistent performance improvements across different database sizes and numbers of incremental backups. Tests will be conducted with varying data volumes and incremental backup chains to ensure scalability.