PS 5.7.25-28 crashed four times in 3 hours

Description

Hello, I have a problem, PSMDB 5.7.25-28 crashed 4 times in 3 hours.

 

 

log:

mysqld: Out of memory (Needed 2123481908 bytes)
2019-10-16T13:57:52.104163+08:00 0 [Note] Stopping ack receiver thread
pure virtual method called
terminate called without an active exception
05:57:52 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at https://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=1048576
max_used_connections=227
max_threads=514
thread_count=106
connection_count=106
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1064052 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fe0b402f110
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fe3700ccd30 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xf142ab]
/usr/sbin/mysqld(handle_fatal_signal+0x471)[0xd4e7a1]
/lib64/libpthread.so.0(+0xf5d0)[0x7fe423c9d5d0]
/lib64/libc.so.6(gsignal+0x37)[0x7fe421d9b207]
/lib64/libc.so.6(abort+0x148)[0x7fe421d9c8f8]
/lib64/libstdc++.so.6(ZN9gnu_cxx27_verbose_terminate_handlerEv+0x165)[0x7fe4226aa9d5]
/lib64/libstdc++.so.6(+0x5e946)[0x7fe4226a8946]
/lib64/libstdc++.so.6(+0x5e973)[0x7fe4226a8973]
/lib64/libstdc++.so.6(+0x5f4df)[0x7fe4226a94df]
/usr/sbin/mysqld(_Z15ha_commit_transP3THDbb+0x131)[0x7adb31]
/usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x32)[0xd1e6d2]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x63a)[0xc6ae6a]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_stateb+0x455)[0xc72195]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xb27)[0xc72da7]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x1df)[0xc7484f]
/usr/sbin/mysqld(handle_connection+0x2c0)[0xd35320]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xf2c514]
/lib64/libpthread.so.0(+0x7dd5)[0x7fe423c95dd5]
/lib64/libc.so.6(clone+0x6d)[0x7fe421e62ead]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fe0b403f6c0): SELECT id,product_id,module_name,module_code FROM module_info WHERE tenant_id='53905' and del_flag=0 AND product_id IN ( 2019-10-16T14:14:32.884240+08:00 0 [Note] /usr/sbin/mysqld (mysqld 5.7.25-28-log) starting as process 91823 ...

.....

2019-10-16T14:14:32.899720+08:00 0 [Note] InnoDB: Setting NUMA memory policy to MPOL_INTERLEAVE
2019-10-16T14:14:43.207411+08:00 0 [Note] InnoDB: Setting NUMA memory policy to MPOL_DEFAULT
2019-10-16T14:14:43.207562+08:00 0 [Note] InnoDB: Completed initialization of buffer pool

...

2019-10-16T14:25:06.845435+08:00 0 [Note] Stopping ack receiver thread
pure virtual method called
terminate called without an active exception
06:25:07 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at https://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=1048576
max_used_connections=161
max_threads=514
thread_count=110
connection_count=110
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1064052 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f14a4012860
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f1587170d30 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xf142ab]
/usr/sbin/mysqld(handle_fatal_signal+0x471)[0xd4e7a1]
/lib64/libpthread.so.0(+0xf5d0)[0x7f1845b4c5d0]
/lib64/libc.so.6(gsignal+0x37)[0x7f1843c4a207]
/lib64/libc.so.6(abort+0x148)[0x7f1843c4b8f8]
/lib64/libstdc++.so.6(ZN9gnu_cxx27_verbose_terminate_handlerEv+0x165)[0x7f18445599d5]
/lib64/libstdc++.so.6(+0x5e946)[0x7f1844557946]
/lib64/libstdc++.so.6(+0x5e973)[0x7f1844557973]
/lib64/libstdc++.so.6(+0x5f4df)[0x7f18445584df]
/usr/sbin/mysqld(_Z40datetime_with_no_zero_in_date_to_timevalP3THDPK13st_mysql_timeP7timevalPi+0xa1)[0xce6e61]
/usr/sbin/mysqld(_ZN33Field_temporal_with_date_and_time25convert_TIME_to_timestampEP3THDPK13st_mysql_timeP7timevalPi+0x1d)[0xdc012d]
/usr/sbin/mysqld(_ZN16Field_timestampf14store_internalEPK13st_mysql_timePi+0x4d)[0xdc026d]
/usr/sbin/mysqld(_ZN24Field_temporal_with_date10store_timeEP13st_mysql_timeh+0x19a)[0xdc9eca]
/usr/sbin/mysqld(_ZN4Item13save_in_fieldEP5Fieldb+0x15)[0x7cb565]
/usr/sbin/mysqld(Z11fill_recordP3THDP5TABLER4ListI4ItemES6_P9st_bitmapS8+0xe6)[0xc14d06]
/usr/sbin/mysqld(_Z36fill_record_n_invoke_before_triggersP3THDP9COPY_INFOR4ListI4ItemES6_P5TABLE23enum_trigger_event_typei+0x2a9)[0xc150e9]
/usr/sbin/mysqld(Z12write_recordP3THDP5TABLEP9COPY_INFOS4+0x8c9)[0xe4b9d9]
/usr/sbin/mysqld(_ZN14Sql_cmd_insert12mysql_insertEP3THDP10TABLE_LIST+0xc33)[0xe4c803]
/usr/sbin/mysqld(_ZN14Sql_cmd_insert7executeEP3THD+0xea)[0xe4cd0a]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0x1869)[0xc6c099]
/usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_stateb+0x455)[0xc72195]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xb27)[0xc72da7]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x1df)[0xc7484f]
/usr/sbin/mysqld(handle_connection+0x2c0)[0xd35320]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xf2c514]
/lib64/libpthread.so.0(+0x7dd5)[0x7f1845b44dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f1843d11ead]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f14a4045d90): INSERT INTO supporter_call_count_date_crm (ea,app_id,url,api_name,call_time,total_count,success_count,fail_count,create_time) VALUES ('ftcxcrm','FSAID_131834c','https://open.fxiaoke.com/cgi/crm/data/create','object_55meR__c','2019-10-16',1,0,1,NOW())ON DUPLICATE KEY UPDATE total_count = total_count + 1,success_count = success_count+0,fail_count = fail_count + 1,create_time = NOW()
Connection ID (thread ID): 580
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

 

2019-10-16T16:47:35.445671+08:00 2661 [Warning] Event 'MYSQL_AUDIT_GENERAL_STATUS' cannot be aborted. The trigger error was (5) [HY000]: Out of memory (Needed 2123481932 bytes)
2019-10-16T16:47:35.446332+08:00 2661 [ERROR] /usr/sbin/mysqld: Out of memory (Needed 2123481932 bytes)
08:47:35 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at https://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=1048576
max_used_connections=198
max_threads=514
thread_count=98
connection_count=98
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1064052 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f5a9806f7d0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f5b1ddf0d30 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xf142ab]
/usr/sbin/mysqld(handle_fatal_signal+0x471)[0xd4e7a1]
/lib64/libpthread.so.0(+0xf5d0)[0x7f5eeed475d0]
/usr/sbin/mysqld(my_convert+0x55)[0x135b145]
/usr/lib64/mysql/plugin/audit_log.so(+0x6960)[0x7f5b257d3960]
/usr/lib64/mysql/plugin/audit_log.so(+0x758d)[0x7f5b257d458d]
/usr/sbin/mysqld(_Z18mysql_audit_notifyP3THD30mysql_event_general_subclass_tPKciS3_m+0x336)[0xd4f596]
/usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x433)[0xc726b3]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x1df)[0xc7484f]
/usr/sbin/mysqld(handle_connection+0x2c0)[0xd35320]
/usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xf2c514]
/lib64/libpthread.so.0(+0x7dd5)[0x7f5eeed3fdd5]
/lib64/libc.so.6(clone+0x6d)[0x7f5eecf0cead]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f5af493e040): UPDATE custom_menu SET gmt_modified=now(),content='{"appId":"FSAID_bec6e8b","menus":[{"actionParam":"https://www.fxiaoke.com/open/imgtext/?appId
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
u0003
Connection ID (thread ID): 2661
Status: NOT_KILLED

You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
2019-10-16T16:47:37.653868+08:00 0 [Note] /usr/sbin/mysqld (mysqld 5.7.25-28-log) starting as process 134544 ...
2019-10-16T16:47:37.661515+08:00 0 [Note] InnoDB: PUNCH HOLE support available
2019-10-16T16:47:37.661548+08:00 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-10-16T16:47:37.661554+08:00 0 [Note] InnoDB: Uses event mutexes
2019-10-16T16:47:37.661559+08:00 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-10-16T16:47:37.661565+08:00 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
2019-10-16T16:47:37.661569+08:00 0 [Note] InnoDB: Using Linux native AIO
2019-10-16T16:47:37.662151+08:00 0 [Note] InnoDB: Number of pools: 1
2019-10-16T16:47:37.662270+08:00 0 [Note] InnoDB: Using CPU crc32 instructions
2019-10-16T16:47:37.666286+08:00 0 [Note] InnoDB: Initializing buffer pool, total size = 14G, instances = 4, chunk size = 128M
2019-10-16T16:47:38.071877+08:00 0 [Note] InnoDB: Completed initialization of buffer pool

 

DB log please See attachment

 

After crashing a few times, I modified a parameter:
Innodb_numa_interleave = 1

to 
Innodb_numa_interleave = 0

I haven’t crashed until today.

 

Please help me to see why crash .. Thank you 

Environment

OS:centos 7 7.3.1611 kernel : 3.10.0-514.26.1.el7.x86_64

memory : 16G  Expansion 24 G  Expansion 30G   ( 2 expansions after crash )

Vcpu : 8

DB data : 29G

Always running stable, it appeared yesterday 。

[xxxx@vlnx022002 data]$ numactl --hardware

available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 30719 MB
node 0 free: 4641 MB
node distances:
node 0
0: 10

 

OS message See attachment

Attachments

3

Smart Checklist

Activity

Show:

max November 11, 2019 at 6:19 AM

Lalit Choudhary November 6, 2019 at 1:33 PM

Hi

Thank you for the report.

 

looking at the crash log mysql crashing due to OMM  mysqld: Out of memory, also I can see you are using the audit log with Percona server 5.7.25 version.

  bug already reported for a similar issue in percona server 5.7.25 and the bug fixed in PS 5.7.26.

Please upgrade your percona server version to the latest version to avoid these crashes. 

https://www.percona.com/downloads/Percona-Server-5.7/LATEST/

max October 21, 2019 at 9:54 AM

VM  info:

DMI: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v1.0 11/26/2012
Hypervisor detected: Microsoft HyperV
HyperV: features 0x2e7f, hints 0x2c2c
HyperV: LAPIC Timer Frequency: 0x30d40
tsc: Marking TSC unstable due to running on Hyper-V
Switched to clocksource hyperv_clocksource

 

Thank you !

Done

Details

Assignee

Reporter

Fix versions

Affects versions

Priority

Smart Checklist

Created October 18, 2019 at 8:49 AM
Updated March 6, 2024 at 11:48 AM
Resolved November 6, 2019 at 1:34 PM
Loading...