With smart memory estimation enabled, Percona Xtrabackup process still used up free memory during prepare stage
General
Escalation
General
Escalation
Description
Summary : While xtrabackup is preparing, OOM occurs.
Affect version : 8.0.32-26
Total memory: 16 G
Description :
Make a backup, which contains 100k tables, every table has 10 partition tables and every partition table has 1k rows. Total data disk usage is around 1T.
Ops platform shows the total memory usage goes up from 0.7% to 99 %, /var/log/message tells xtrabackup process was killed by oom-killer. Total prepare time before process killed is 30mins.
OOM also occurs while using option --use-memory, it seems both of them can't limit memory used by xtrabackup. I guess xtrabackup loading all tablespace to memory may cause this overuse of memory.
In the prepare phase, Percona XtraBackup checks the server’s available free memory and uses that memory up to the limit specified in the --use-free-memory-pct option to run --prepare. Due to backward compatibility, the default value for the --use-free-memory-pct option is 0 (zero), which defines the option as disabled. For example, if you set --use-free-memory-pct=50, then 50% of the free memory is used to prepare a backup.
Environment
OS: CentOS7
Total memory: 16G
Activity
Marcelo Altmann October 23, 2023 at 9:37 AM
Hi @Zc .
Can you please check the same scenario with Xtrabackup 8.0.33-28. We have implemented https://jira.percona.com/browse/PXB-2955 which reduces the amount of memory required during prepare.
Summary : While xtrabackup is preparing, OOM occurs.
Affect version : 8.0.32-26
Total memory: 16 G
Description :
Make a backup, which contains 100k tables, every table has 10 partition tables and every partition table has 1k rows.
Total data disk usage is around 1T.
Command:
xtrabackup --use-free-memory-pct=50 --estimate-memory=ON --prepare ...Ops platform shows the total memory usage goes up from 0.7% to 99 %, /var/log/message tells xtrabackup process was killed by oom-killer. Total prepare time before process killed is 30mins.
OOM also occurs while using option --use-memory, it seems both of them can't limit memory used by xtrabackup. I guess xtrabackup loading all tablespace to memory may cause this overuse of memory.
https://docs.percona.com/percona-xtrabackup/8.0/smart-memory-estimation.html?h=smart#how-does-smart-memory-estimation-work
percona docs tells:
In the
preparephase, Percona XtraBackup checks the server’s available free memory and uses that memory up to the limit specified in the--use-free-memory-pctoption to run--prepare. Due to backward compatibility, the default value for the--use-free-memory-pctoption is 0 (zero), which defines the option as disabled. For example, if you set--use-free-memory-pct=50, then 50% of the free memory is used topreparea backup.