LP #1582456: Incremental restore in XB 2.4 requires 1+GB of RAM
Description
Environment
Activity
**Comment from Launchpad by: Sergei Glushchenko on: 11-07-2016 06:54:53
**Comment from Launchpad by: Andrey Galkin on: 09-07-2016 17:54:54
Hi Sergei,
Yes, I understood it was upstream change. Perhaps, you may want to use mmap'ed file to workaround the case. If there is enough memory then OS would operate in plain RAM with minimal overhead as before. Low memory systems would slow down, but still continue to operate instead of hard failing.
Thanks for looking into that.
**Comment from Launchpad by: Sergei Glushchenko on: 09-07-2016 16:55:30
Hello Andrey,
MySQL starting with 5.7.6 supports page size up to 64K, so
`UNIV_PAGE_SIZE_MAX' is the upstream change. It would be not trivial to
change amount of memory needed for `incremental_buffer' without changing
`.delta' file format.
So, for the larger fix I started to draft this blueprint:
https://blueprints.launchpad.net/percona-xtrabackup/+spec/incremental-memory-limit
What I am going to do now is to replace `UNIV_PAGE_SIZE_MAX' with
`page_size', so for default page size we will still use 64M buffer.
**Comment from Launchpad by: Andrey Galkin on: 24-05-2016 03:36:30
It may not look like important for production. However, it has implications for test automation activities involving XB.
Please consider returning memory usage back to fixed 64MB for incremental buffer at least.
Details
Details
Assignee
Reporter
Labels
Priority
High
**Reported in Launchpad by Andrey Galkin last update 11-07-2016 13:12:21
Steps to reproduce:
------------------- S1. Create a low memory VM e.g. <1GB (alternatively, limit with ulimit/cgroup)
S2. Make sure swap is off
S3. Create full backup + incremental backup
S4. Try to restore
S5. See a failure on incremental:
...
Applying /db/mysql_mysrv1/data.incremental/ibdata1.delta to ./ibdata1...
InnoDB: Retry attempts for reading partial data failed.
InnoDB: Tried to read 16384 bytes at offset 0 was only able to read0
InnoDB: Operating system error number 14 in a file operation.
InnoDB: Error number 14 means 'Bad address'
...
What is expected:
----------------- W1. Make sure out-of-memory error is properly reported (check malloc() return)
W2. Use smaller buffer of 64MB as before or even less, do not use UNIV_PAGE_SIZE_MAX in calculations
W3. Obey --use-memory as much as possible
Debug results:
-------------- D1. xtrabackup.cc:5844 (https://github.com/percona/percona-xtrabackup/blob/2.4/storage/innobase/xtrabackup/src/xtrabackup.cc#L5844)
...
incremental_buffer_base = static_cast<byte *>
(ut_malloc_nokey((UNIV_PAGE_SIZE_MAX / 4 + 1) *
UNIV_PAGE_SIZE_MAX));
incremental_buffer = static_cast<byte *>
(ut_align(incremental_buffer_base,
UNIV_PAGE_SIZE_MAX));
...
D2. incremental_buffer_base & incremental_buffer are NOT checked other than for ut_free()
D3. UNIV_PAGE_SIZE_MAX has changed due to change of UNIV_PAGE_SIZE_SHIFT_MAX from 14 to 16 in rev 717b4b2e
#define UNIV_PAGE_SIZE_MAX (1 << UNIV_PAGE_SIZE_SHIFT_MAX)
D4. Consequences memory usage for the buffer raised from ~64MB to ~1GB