Strange memory management when using full-text indexes

It appears that this issue is no longer being worked on, so we are closing it for housekeeping purposes. If you believe the issue still exists, please open a new ticket after confirming it's present in the latest release.

FYI: blogpost link:

Idea (2) - jemalloc works like a charm

Here are a few few ideas, which we can verify.
1. Pre-allocating memory_heap object inside 'fts_query_phrase_search()' with a big memory block(32/64/128 MB) upon creation.
2. Try different memory allocator like jemalloc (may have other side effects).
3. There can be kernel tuning settings for memory releasing background thread (needs investigation).

Deeper InnoDB FTS code analysis and checks revealed no actual memory leaks.

The problem seems to be in memory fragmentation and in how this fragmented memory is released to the OS upon calling 'free()' libc function.

Each call to 'fts_query_phrase_search()' creates an InnoDB memory heap object which grows up to 80 MB in this case and have a big number of memory blocks ('mem_block_t'). Memory in those blocks is very often wasted as this heap is used for constantly growing 'ib_vector' objects and every time 'ib_vector_push()' is called and vector runs out of its capacity, a new memory is requested from the heap_allocator which in turns ends up with allocating a new memory block for the heap by calling 'malloc()'.

Digging into how standard libc memory allocator works, it looks like 'free()' function MAY but DON'T HAVE to immediately release memory to the kernel. It may reuse the most recently deleted block (or several blocks) for the next allocation(s). Moreover, releasing memory to the kernel is still just an async request. On modern kernels, the memory is actually released by a background thread. The fact that having 2 CPU cores instead of 1 drastically improves memory consumption can be an explained by the fact that in this case that memory releasing background thread now can be run on its own CPU core and be much more productive and therefore catch up with allocation requests coming from mysqld.

See https://unix.stackexchange.com/questions/53447/does-free-unmap-the-memory-of-a-process).