Correct the buffer calculation for the audit plugin used when large queries are executed(PS-5395).
Description
Environment
AFFECTED CS IDs
Attachments
clones
Smart Checklist
Activity

Marcelo Altmann October 14, 2020 at 2:46 PMEdited
Function audit_log_general_record can be called twice for each record.
1st time it is called with a 4096k buffer. It will estimate via
query_length what can be the maximum size of incoming query if every
character has to be converted to utf8mb4(basically 4 times its size).
If this size fits the buffer, it will convert event->general_query.str
to utf8mb4, use the current buffer to store the converted query and
advance the return pointer by query_length bytes. After we have the real
query, we escape it.
Otherwise (if the buffer is not big enough to convert the query), we
will fall into the else branch. Here we calculate the size of the
current query after escaping it and multiply by 4 (since we have not
converted the query to utf8mb4) + the maximum size of incoming query if
every character has to be converted to utf8mb4.
We continue by escaping user / host / external_user / ip and db storing
the required bytes into buflen_estimated.
After this point we do a check to validate if the current buffer is big
enough to hold buflen_estimated (the converted query + the full output
of audit log entry). In case we don't have a big enough buffer, we set
outlen to buflen_estimated and return NULL. Then audit_log_notify will
allocate outlen bytes into its buffer and call audit_log_general_record
for the 2nd time, this time with the resized buffer.
Issue:
Before 9fd57d3
we estimated buflen_estimated as twice of the output. If
it was not enough we would increase the size of buffer by 4 times before
calling audit_log_general_record the second time.
After 9fd57d3
the else branch has been enhance to make a better estimate
of the size of the output + size of the query, taking into consideration
utf8mb4, however it missed two points:
buflen_estimated was doubling the size. This is no longer required
since the else branch already take into account escaped utf8mb4 query +
non escaped utf8mb4 query.When estimating the size of the non escaped utf8mb4 query, it was
wrongly multiplying its size by my_charset_utf8mb4_general_ci.mbmaxlen,
however, this computation has already been performed before the if/else.
Fix:
Adjusted the buffer calculation to not allocate unnecessary memory.

Sergey Kuzmichev September 28, 2020 at 9:31 AMEdited
Attached bcc/memleak output as
and jemalloc's profiling visualized as
. Both agree on the code path leading to the allocation. Tools were executed while running modified test case (100MiB blob instead of 10MiB). memleak omits allocations <1MiB in size.
plugin_var_memalloc_session_update
will account for memory in P_S under memory/sql/THD::variables
, see https://github.com/percona/percona-server/blob/5.7/sql/sql_plugin.cc#L3563

Jira Bot September 26, 2020 at 3:56 PM
To: Former user
CC:
Hi, I'm jira-bot, Percona's Jira automation tool. I've detected that someone from
Percona has made an edit to the Summary field of an issue that you reported.
I'm not sentient (yet) so I'm not sure whether the person fixed a typo, changed
a few words, or completely rewrote the text. In any case, it is Percona Engineering's
intention to make the Summary and Description of an issue as accurate as possible
so that we're fixing the actual problem you're encountering, and to avoid
misunderstandings about symptoms and causes.
If the current Summary does not accurately reflect the problem you are reporting,
or if you feel the change was otherwise inappropriate in some way, please add a
new comment explaining things and we'll address it as soon as we can.
This message will be added only once per issue, regardless of how many times
the Summary is edited.
message-code:summary-edited

Lalit Choudhary September 26, 2020 at 3:41 PM
Also reproducible with PXC.
With a low RAM machine, this resulting in OOM crash.
mysql crash logs:
test: PS 5.7.31
Disabling the audit log plugin will not increase memory for the same test.
Details
Details
Assignee

Reporter

Labels
Needs Review
Time tracking
Fix versions
Priority
Smart Checklist
Open Smart Checklist
Smart Checklist

Same issue as: https://jira.percona.com/browse/PS-5395
When audit_plugin is enabled MySQL uses a lot of memory when handling large queries.
Steps to reproduce:
Install audit plugin
Test-case:
Create schema:
Memory usage when audit plugin is not enabled:
Memory usage when the audit plugin is enabled