mysqld got signal 11
Description
Environment
Smart Checklist
Activity
Dmitry November 28, 2018 at 7:56 AM
Thank you, @Lalit Choudhary
— Here from what version you are moved from MySQL to Percona server is an important thing.
It was mysql-5.6.32 as I remember. Hardware and OS are the same.
— It would be good if you run mysql_upgrade which fix table structure differences and corrupt table problems.
Yes, I'll do that when it can be possible and monitor situation.
— Percona has it's own Audit_Log Plugin, tested with Percona server, use of it will be good to avoid problems.
Yes, we use MariaDB audit plugin. It worked without any problems with "vanilla" mysql-5.6. I know about percona audit and will test it ASAP.
Lalit Choudhary November 12, 2018 at 1:11 PM
Hi Dmitry,
Thank you for information.
I did test this issue on Percona Server 5.6.41 with https://perconadev.atlassian.net/browse/PS-1085#icft=PS-1085 test case which has similar backtrace crash. I cannot reproduce described behavior.
> Did you run mysql_upgrade after replacing the binaries?
No. Simply removed MySQL and installed Percona Server. Possible tables "mismatches and errors"?
Same way (removing mysql -> installing percona) I migrated from mysql to percona on slave server - it runs OK.
Here from what version you are moved from MySQL to Percona server is an important thing.
It would be good if you run mysql_upgrade which fix table structure differences and corrupt table problems.
Also, In the provided log I see following,
181108 2:23:49 server_audit: MariaDB Audit Plugin version 1.4.0 STARTED.
Percona has it's own Audit_Log Plugin, tested with Percona server, use of it will be good to avoid problems.
https://www.percona.com/doc/percona-server/5.7/management/audit_log_plugin.html
After fixing this, if you get crash then update this bug with crash error info.
Dmitry November 8, 2018 at 3:07 PM
Thank you for answer.
> Did you run mysql_upgrade after replacing the binaries?
No. Simply removed MySQL and installed Percona Server. Possible tables "mismatches and errors"?
Same way (removing mysql -> installing percona) I migrated from mysql to percona on slave server - it runs OK.
> Also, Provide full mysql error log for crash instance
This was full error log. Also, at 01:00 AM some complicate srcipt started and crashed mysql after 1,5 hours of working with next errors:
2018-11-08 02:23:42 29128 [Note] Plugin 'FEDERATED' is disabled.
2018-11-08 02:23:43 29128 [ERROR] Can't open shared library '/usr/lib/mysql/plugin/libaudit_plugin.so' (errno: 0 /usr/lib/mysql/plugin/libaudit_plugin.so
2018-11-08 02:23:43 29128 [Warning] Couldn't load plugin named 'MCAFEE' with soname 'libaudit_plugin.so'.
2018-11-08 02:23:43 29128 [ERROR] Function 'server_audit' already exists
2018-11-08 02:23:43 29128 [Warning] Couldn't load plugin named 'server_audit' with soname 'server_audit.so'.
2018-11-08 02:23:43 29128 [Note] InnoDB: Using atomics to ref count buffer pool pages
2018-11-08 02:23:43 29128 [Note] InnoDB: The InnoDB memory heap is disabled
2018-11-08 02:23:43 29128 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-11-08 02:23:43 29128 [Note] InnoDB: Memory barrier is not used
2018-11-08 02:23:43 29128 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-11-08 02:23:43 29128 [Note] InnoDB: Using Linux native AIO
2018-11-08 02:23:43 29128 [Note] InnoDB: Using CPU crc32 instructions
2018-11-08 02:23:43 29128 [Note] InnoDB: Initializing buffer pool, size = 25.0G
2018-11-08 02:23:44 29128 [Note] InnoDB: Completed initialization of buffer pool
2018-11-08 02:23:44 29128 [Note] InnoDB: Highest supported file format is Barracuda.
2018-11-08 02:23:44 29128 [Note] InnoDB: The log sequence numbers 24284754405431 and 24284754405431 in ibdata files do not match the log sequence number
2018-11-08 02:23:44 29128 [Note] InnoDB: Database was not shutdown normally!
2018-11-08 02:23:44 29128 [Note] InnoDB: Starting crash recovery.
2018-11-08 02:23:44 29128 [Note] InnoDB: Reading tablespace information from the .ibd files...
2018-11-08 02:23:47 29128 [Note] InnoDB: Restoring possible half-written data pages.
2018-11-08 02:23:47 29128 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Last MySQL binlog file position 0 28703, file name mysql-bin.000151
2018-11-08 02:23:49 29128 [Note] InnoDB: 128 rollback segment(s) are active.
2018-11-08 02:23:49 29128 [Note] InnoDB: Waiting for purge to start
2018-11-08 02:23:49 29128 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.41-84.1 started; log sequence number 24457337590681
181108 2:23:49 server_audit: server_audit_incl_users set to 'D14_Bakushin'.
181108 2:23:49 server_audit: MariaDB Audit Plugin version 1.4.0 STARTED.
181108 2:23:49 server_audit: logging started to the file /var/log/mysql/bide-audit.log.
2018-11-08 02:23:49 29128 [Note] Recovering after a crash using /var/log/mysql/mysql-bin
2018-11-08 02:23:49 29128 [Note] Starting crash recovery...
2018-11-08 02:23:49 29128 [Note] Crash recovery finished.
2018-11-08 02:23:50 29128 [Note] RSA private key file not found: /mysql-data//private_key.pem. Some authentication plugins will not work.
2018-11-08 02:23:50 29128 [Note] RSA public key file not found: /mysql-data//public_key.pem. Some authentication plugins will not work.
2018-11-08 02:23:50 29128 [Note] Server hostname (bind-address): '*'; port: 3306
2018-11-08 02:23:50 29128 [Note] IPv6 is available.
2018-11-08 02:23:50 29128 [Note] - '::' resolves to '::';
2018-11-08 02:23:50 29128 [Note] Server socket created on IP: '::'.
2018-11-08 02:23:50 29128 [Warning] 'user' entry 'root@bide' ignored in --skip-name-resolve mode.
2018-11-08 02:23:50 29128 [Warning] 'proxies_priv' entry '@ root@bide' ignored in --skip-name-resolve mode.
2018-11-08 02:23:50 29128 [Note] Event Scheduler: Loaded 1 event
2018-11-08 02:23:50 29128 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.41-84.1-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona Server (GPL), Release 84.1, Revision b308619
2018-11-08 02:23:50 29128 [Note] Event Scheduler: scheduler thread started with id 1
> Please share event definition and definition of all object included in it like store procedure and tables it referring to.
mysql> select name from mysql.proc;
mysql> select name from mysql.proc;
+-------------------------------------+
| name |
+-------------------------------------+
| AllInsp |
| lk_esvg |
| lk_esvg_1 |
| lk_patent_fizlic |
| lk_patent_fizlic_1 |
| lk_prazdn |
| lk_prazdn_minus |
| RUB |
| create_synonym_db |
| extract_schema_from_file_name |
| extract_table_from_file_name |
| format_bytes |
| format_path |
| format_statement |
| format_time |
| ps_is_account_enabled |
| ps_is_consumer_enabled |
| ps_is_instrument_default_enabled |
| ps_is_instrument_default_timed |
| ps_is_thread_instrumented |
| ps_setup_disable_background_threads |
| ps_setup_disable_consumer |
| ps_setup_disable_instrument |
| ps_setup_disable_thread |
| ps_setup_enable_background_threads |
| ps_setup_enable_consumer |
| ps_setup_enable_instrument |
| ps_setup_enable_thread |
| ps_setup_reload_saved |
| ps_setup_reset_to_default |
| ps_setup_save |
| ps_setup_show_disabled |
| ps_setup_show_disabled_consumers |
| ps_setup_show_disabled_instruments |
| ps_setup_show_enabled |
| ps_setup_show_enabled_consumers |
| ps_setup_show_enabled_instruments |
| ps_statement_avg_latency_histogram |
| ps_thread_id |
| ps_thread_stack |
| ps_trace_statement_digest |
| ps_trace_thread |
| ps_truncate_all_tables |
| sys_get_config |
+-------------------------------------+
Our main database is 'analiz_0'. So, if filter according to main database we get only 6 rows:
mysql> show procedure status where Db='analiz_0'\G
*************************** 1. row ***************************
Db: analiz_0
Name: AllInsp
Type: PROCEDURE
Definer: webuser@localhost
Modified: 2017-10-06 11:38:12
Created: 2017-10-06 11:38:12
Security_type: DEFINER
Comment:
character_set_client: utf8
collation_connection: utf8_general_ci
Database Collation: utf8_general_ci
*************************** 2. row ***************************
Db: analiz_0
Name: lk_esvg
Type: PROCEDURE
Definer: webuser@127.0.0.1
Modified: 2017-11-15 12:52:28
Created: 2017-11-15 12:52:28
Security_type: DEFINER
Comment:
character_set_client: utf8
collation_connection: utf8_general_ci
Database Collation: utf8_general_ci
*************************** 3. row ***************************
Db: analiz_0
Name: lk_esvg_1
Type: PROCEDURE
Definer: webuser@127.0.0.1
Modified: 2017-11-14 10:46:42
Created: 2017-11-14 10:46:42
Security_type: DEFINER
Comment:
character_set_client: utf8
collation_connection: utf8_general_ci
Database Collation: utf8_general_ci
*************************** 4. row ***************************
Db: analiz_0
Name: lk_patent_fizlic
Type: PROCEDURE
Definer: webuser@127.0.0.1
Modified: 2017-11-15 12:51:43
Created: 2017-11-15 12:51:43
Security_type: DEFINER
Comment:
character_set_client: utf8
collation_connection: utf8_general_ci
Database Collation: utf8_general_ci
*************************** 5. row ***************************
Db: analiz_0
Name: lk_patent_fizlic_1
Type: PROCEDURE
Definer: webuser@127.0.0.1
Modified: 2017-11-14 10:43:32
Created: 2017-11-14 10:43:32
Security_type: DEFINER
Comment:
character_set_client: utf8
collation_connection: utf8_general_ci
Database Collation: utf8_general_ci
*************************** 6. row ***************************
Db: analiz_0
Name: RUB
Type: PROCEDURE
Definer: root@localhost
Modified: 2016-09-23 13:01:49
Created: 2016-09-23 13:01:49
Security_type: DEFINER
Comment:
character_set_client: utf8
collation_connection: utf8_general_ci
Database Collation: utf8_general_ci
There're complicate SQLs with hundreds or even thousands of queries. Pasting them here will be like a spamming.
> Provide SQL command and SHOW CREATE TABLE output for tables included in Store procedure
Same thing: too many information will be. Too many...
Maybe there's something else, more concrete to see and paste here?
Dmitry November 8, 2018 at 1:38 PM
Thank you for answer.
> Did you run mysql_upgrade after replacing the binaries?
No. Simply removed MySQL and installed Percona Server. Possible tables "mismatches and errors"?
Same way (removing mysql -> installing percona) I migrated from mysql to percona on slave server - it runs OK.
> Also, Provide full mysql error log for crash instance
This was full error log. Also, at 01:00 AM some complicate srcipt started and crashed mysql after 1,5 hours of working with next errors:
2018-11-08 02:23:42 29128 [Note] Plugin 'FEDERATED' is disabled.
2018-11-08 02:23:43 29128 [ERROR] Can't open shared library '/usr/lib/mysql/plugin/libaudit_plugin.so' (errno: 0 /usr/lib/mysql/plugin/libaudit_plugin.so
2018-11-08 02:23:43 29128 [Warning] Couldn't load plugin named 'MCAFEE' with soname 'libaudit_plugin.so'.
2018-11-08 02:23:43 29128 [ERROR] Function 'server_audit' already exists
2018-11-08 02:23:43 29128 [Warning] Couldn't load plugin named 'server_audit' with soname 'server_audit.so'.
2018-11-08 02:23:43 29128 [Note] InnoDB: Using atomics to ref count buffer pool pages
2018-11-08 02:23:43 29128 [Note] InnoDB: The InnoDB memory heap is disabled
2018-11-08 02:23:43 29128 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-11-08 02:23:43 29128 [Note] InnoDB: Memory barrier is not used
2018-11-08 02:23:43 29128 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-11-08 02:23:43 29128 [Note] InnoDB: Using Linux native AIO
2018-11-08 02:23:43 29128 [Note] InnoDB: Using CPU crc32 instructions
2018-11-08 02:23:43 29128 [Note] InnoDB: Initializing buffer pool, size = 25.0G
2018-11-08 02:23:44 29128 [Note] InnoDB: Completed initialization of buffer pool
2018-11-08 02:23:44 29128 [Note] InnoDB: Highest supported file format is Barracuda.
2018-11-08 02:23:44 29128 [Note] InnoDB: The log sequence numbers 24284754405431 and 24284754405431 in ibdata files do not match the log sequence number
2018-11-08 02:23:44 29128 [Note] InnoDB: Database was not shutdown normally!
2018-11-08 02:23:44 29128 [Note] InnoDB: Starting crash recovery.
2018-11-08 02:23:44 29128 [Note] InnoDB: Reading tablespace information from the .ibd files...
2018-11-08 02:23:47 29128 [Note] InnoDB: Restoring possible half-written data pages.
2018-11-08 02:23:47 29128 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Last MySQL binlog file position 0 28703, file name mysql-bin.000151
2018-11-08 02:23:49 29128 [Note] InnoDB: 128 rollback segment(s) are active.
2018-11-08 02:23:49 29128 [Note] InnoDB: Waiting for purge to start
2018-11-08 02:23:49 29128 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.41-84.1 started; log sequence number 24457337590681
181108 2:23:49 server_audit: server_audit_incl_users set to 'D14_Bakushin'.
181108 2:23:49 server_audit: MariaDB Audit Plugin version 1.4.0 STARTED.
181108 2:23:49 server_audit: logging started to the file /var/log/mysql/bide-audit.log.
2018-11-08 02:23:49 29128 [Note] Recovering after a crash using /var/log/mysql/mysql-bin
2018-11-08 02:23:49 29128 [Note] Starting crash recovery...
2018-11-08 02:23:49 29128 [Note] Crash recovery finished.
2018-11-08 02:23:50 29128 [Note] RSA private key file not found: /mysql-data//private_key.pem. Some authentication plugins will not work.
2018-11-08 02:23:50 29128 [Note] RSA public key file not found: /mysql-data//public_key.pem. Some authentication plugins will not work.
2018-11-08 02:23:50 29128 [Note] Server hostname (bind-address): '*'; port: 3306
2018-11-08 02:23:50 29128 [Note] IPv6 is available.
2018-11-08 02:23:50 29128 [Note] - '::' resolves to '::';
2018-11-08 02:23:50 29128 [Note] Server socket created on IP: '::'.
2018-11-08 02:23:50 29128 [Warning] 'user' entry 'root@bide' ignored in --skip-name-resolve mode.
2018-11-08 02:23:50 29128 [Warning] 'proxies_priv' entry '@ root@bide' ignored in --skip-name-resolve mode.
2018-11-08 02:23:50 29128 [Note] Event Scheduler: Loaded 1 event
2018-11-08 02:23:50 29128 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.41-84.1-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona Server (GPL), Release 84.1, Revision b308619
2018-11-08 02:23:50 29128 [Note] Event Scheduler: scheduler thread started with id 1
> Please share event definition and definition of all object included in it like store procedure and tables it referring to.
mysql> select name from mysql.proc;
gives 44 rows.
select name from mysql.proc\G
*************************** 1. row ***************************
name: AllInsp
*************************** 2. row ***************************
name: lk_esvg
*************************** 3. row ***************************
name: lk_esvg_1
*************************** 4. row ***************************
name: lk_patent_fizlic
*************************** 5. row ***************************
name: lk_patent_fizlic_1
*************************** 6. row ***************************
name: lk_prazdn
*************************** 7. row ***************************
name: lk_prazdn_minus
*************************** 8. row ***************************
name: RUB
*************************** 9. row ***************************
name: create_synonym_db
*************************** 10. row ***************************
name: extract_schema_from_file_name
*************************** 11. row ***************************
name: extract_table_from_file_name
*************************** 12. row ***************************
name: format_bytes
*************************** 13. row ***************************
name: format_path
*************************** 14. row ***************************
name: format_statement
*************************** 15. row ***************************
name: format_time
*************************** 16. row ***************************
name: ps_is_account_enabled
*************************** 17. row ***************************
name: ps_is_consumer_enabled
*************************** 18. row ***************************
name: ps_is_instrument_default_enabled
*************************** 19. row ***************************
name: ps_is_instrument_default_timed
*************************** 20. row ***************************
name: ps_is_thread_instrumented
*************************** 21. row ***************************
name: ps_setup_disable_background_threads
*************************** 22. row ***************************
name: ps_setup_disable_consumer
*************************** 23. row ***************************
name: ps_setup_disable_instrument
*************************** 24. row ***************************
name: ps_setup_disable_thread
*************************** 25. row ***************************
name: ps_setup_enable_background_threads
*************************** 26. row ***************************
name: ps_setup_enable_consumer
*************************** 27. row ***************************
name: ps_setup_enable_instrument
*************************** 28. row ***************************
name: ps_setup_enable_thread
*************************** 29. row ***************************
name: ps_setup_reload_saved
*************************** 30. row ***************************
name: ps_setup_reset_to_default
*************************** 31. row ***************************
name: ps_setup_save
*************************** 32. row ***************************
name: ps_setup_show_disabled
*************************** 33. row ***************************
name: ps_setup_show_disabled_consumers
*************************** 34. row ***************************
name: ps_setup_show_disabled_instruments
*************************** 35. row ***************************
name: ps_setup_show_enabled
*************************** 36. row ***************************
name: ps_setup_show_enabled_consumers
*************************** 37. row ***************************
name: ps_setup_show_enabled_instruments
*************************** 38. row ***************************
name: ps_statement_avg_latency_histogram
*************************** 39. row ***************************
name: ps_thread_id
*************************** 40. row ***************************
name: ps_thread_stack
*************************** 41. row ***************************
name: ps_trace_statement_digest
*************************** 42. row ***************************
name: ps_trace_thread
*************************** 43. row ***************************
name: ps_truncate_all_tables
*************************** 44. row ***************************
name: sys_get_config
Our main database is 'analiz_0'. So, if filter according to main database we get only 6 rows:
mysql> show procedure status where Db='analiz_0'\G
*************************** 1. row ***************************
Db: analiz_0
Name: AllInsp
Type: PROCEDURE
Definer: webuser@localhost
Modified: 2017-10-06 11:38:12
Created: 2017-10-06 11:38:12
Security_type: DEFINER
Comment:
character_set_client: utf8
collation_connection: utf8_general_ci
Database Collation: utf8_general_ci
*************************** 2. row ***************************
Db: analiz_0
Name: lk_esvg
Type: PROCEDURE
Definer: webuser@127.0.0.1
Modified: 2017-11-15 12:52:28
Created: 2017-11-15 12:52:28
Security_type: DEFINER
Comment:
character_set_client: utf8
collation_connection: utf8_general_ci
Database Collation: utf8_general_ci
*************************** 3. row ***************************
Db: analiz_0
Name: lk_esvg_1
Type: PROCEDURE
Definer: webuser@127.0.0.1
Modified: 2017-11-14 10:46:42
Created: 2017-11-14 10:46:42
Security_type: DEFINER
Comment:
character_set_client: utf8
collation_connection: utf8_general_ci
Database Collation: utf8_general_ci
*************************** 4. row ***************************
Db: analiz_0
Name: lk_patent_fizlic
Type: PROCEDURE
Definer: webuser@127.0.0.1
Modified: 2017-11-15 12:51:43
Created: 2017-11-15 12:51:43
Security_type: DEFINER
Comment:
character_set_client: utf8
collation_connection: utf8_general_ci
Database Collation: utf8_general_ci
*************************** 5. row ***************************
Db: analiz_0
Name: lk_patent_fizlic_1
Type: PROCEDURE
Definer: webuser@127.0.0.1
Modified: 2017-11-14 10:43:32
Created: 2017-11-14 10:43:32
Security_type: DEFINER
Comment:
character_set_client: utf8
collation_connection: utf8_general_ci
Database Collation: utf8_general_ci
*************************** 6. row ***************************
Db: analiz_0
Name: RUB
Type: PROCEDURE
Definer: root@localhost
Modified: 2016-09-23 13:01:49
Created: 2016-09-23 13:01:49
Security_type: DEFINER
Comment:
character_set_client: utf8
collation_connection: utf8_general_ci
Database Collation: utf8_general_ci
There're complicate SQLs with hundreds or even thousands of queries. Pasting them here will be like a spamming.
> Provide SQL command and SHOW CREATE TABLE output for tables included in Store procedure
Same thing: too many information will be. Too many...
Maybe there's something else, more concrete to see and paste here?
Lalit Choudhary November 8, 2018 at 11:59 AM
Hi Dmitry,
Thank you for report and information.
The crash log has an event execution and procedure call. As you mentioned there's no load on the server (scripts, user access, backups etc.) at the time of crash then this crash could be triggered by an event call.
Please share event definition and definition of all object included in it like store procedure and tables it referring to.
Provide SQL command and SHOW CREATE TABLE output for tables included in Store procedure.
Also, Provide full mysql error log for crash instance.
backtrace looks similar to https://jira.percona.com/browse/PS-1085
>Previously MySQL-5.6 was used, then it was removed and Percona Server was installed up over existing database files.
Did you run mysql_upgrade after replacing the binaries?
Details
Assignee
UnassignedUnassignedReporter
DmitryDmitry(Deactivated)Affects versions
Priority
High
Details
Details
Assignee
Reporter
Affects versions
Priority
Smart Checklist
Open Smart Checklist
Smart Checklist
Open Smart Checklist
Smart Checklist

Debian-9.5, Percona Server 5.6.41-84.1 (Percona Server (GPL), Release 84.1, Revision b308619) running on HP DL180G6 (2x Xeon E5645 @ 2.40GHz) 48 Gb RAM, 4x SAS HDD HP 600GB 15K in RAID1 (for mysql-data)
Sudden crush occurs with next error:
23:23:38 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,.
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/
key_buffer_size=104857600
read_buffer_size=33554432
max_used_connections=367
max_threads=402
thread_count=5
connection_count=3
It is possible that mysqld could use up to.
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 26453457 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7fe6c4004520
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fe6e0d33daf thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x562374e33f4c]
/usr/sbin/mysqld(handle_fatal_signal+0x481)[0x562374bbc801]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7fedde8810c0]
/usr/sbin/mysqld(+0x6e6440)[0x562374e6e440]
/usr/sbin/mysqld(_ZN17Query_cache_query16unlock_n_destroyEv+0x27)[0x562374bfa867]
/usr/sbin/mysqld(ZN11Query_cache12move_by_typeEPPhPP17Query_cache_blockPmS3+0x3f1)[0x562374bfdf91]
/usr/sbin/mysqld(_ZN11Query_cache10pack_cacheEv+0x71)[0x562374bfe4c1]
/usr/sbin/mysqld(_ZN11Query_cache4packEmj+0x44)[0x562374bfe7d4]
/usr/sbin/mysqld(_Z20reload_acl_and_cacheP3THDmP10TABLE_LISTPi+0x3e0)[0x562374c57810]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4750)[0x562374c3f020]
/usr/sbin/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x53)[0x562374d69eb3]
/usr/sbin/mysqld(_ZN12sp_lex_instr23reset_lex_and_exec_coreEP3THDPjb+0x29c)[0x562374d6c1fc]
/usr/sbin/mysqld(_ZN12sp_lex_instr29validate_lex_and_execute_coreEP3THDPjb+0xaa)[0x562374d6cb3a]
/usr/sbin/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x2f8)[0x562374d6cfd8]
/usr/sbin/mysqld(_ZN7sp_head7executeEP3THDb+0x463)[0x562374bc24a3]
/usr/sbin/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x656)[0x562374bc56a6]
/usr/sbin/mysqld(_ZN14Event_job_data7executeEP3THDb+0x8c4)[0x562374cdd674]
/usr/sbin/mysqld(_ZN19Event_worker_thread3runEP3THDP28Event_queue_element_for_exec+0xe5)[0x562374db2665]
/usr/sbin/mysqld(event_worker_thread+0x4d)[0x562374db271d]
/usr/sbin/mysqld(pfs_spawn_thread+0x146)[0x562374e6b4e6]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7494)[0x7fedde877494]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7feddc7a8acf]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fe524071dd0): is an invalid pointer
Connection ID (thread ID): 1116891
Status: NOT_KILLED
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
The time, when crash happens, there's no load on server (scripts, user access, backups etc.)
my.cnf:
===
[mysqld]
port = 3306
connect_timeout=100
net_read_timeout=90
net_write_timeout=120
max_connections = 400
max-connect-errors = 1000
skip-name-resolve
open_files_limit = 100000
datadir = /mysql-data
tmpdir = /ssd
join_buffer_size = 32M
sort_buffer_size = 32M
read_buffer_size = 32M
read_rnd_buffer_size = 32M
key_buffer_size = 100M
max_allowed_packet = 512M
query_cache_size = 64M
query_cache_type = ON
socket = /var/run/mysqld/mysqld.sock
table_open_cache = 10000
thread_cache_size = 32
tmp_table_size = max_heap_table_size
max_heap_table_size = 80M
tmp_table_size = 80M
Replication #####
server-id = 1
binlog-format = MIXED
log_bin= /var/log/mysql/mysql-bin.log
expire_logs_days= 5
max_binlog_size= 1024M
End replication #####
Logs #####
expire_logs_days=7
sync_binlog = 0
#log_slow_queries = /var/log/mysql/bide-slow.log
#long_query_time = 10
End Logs #####
InnoDB #####
innodb_file_per_table
innodb_flush_method = O_DIRECT
50-80% of memory
innodb_buffer_pool_size = 25G
innodb_log_buffer_size = 16M
Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 900M
innodb_flush_log_at_trx_commit = 2
innodb_buffer_pool_instances = 30
interactive_timeout = 21600
wait_timeout = 7200
innodb_lock_wait_timeout = 200
innodb_read_io_threads = 8
innodb_write_io_threads = 8
(numCPU) x 2 + numHDD
innodb_thread_concurrency = 16
innodb_adaptive_hash_index = OFF
innodb_io_capacity = 1500
innodb_io_capacity_max = 2000
End InnoDB #####
event_scheduler=ON
Audit plugin #####
load plugin
plugin-load=server_audit=server_audit.so
only audit connections and DDL queries
server_audit_events=QUERY
enable logging
server_audit_logging=ON
server_audit_incl_users='D14_Bakushin'
flat file
server_audit_output_type=FILE
server_audit_file_path=/var/log/mysql/bide-audit.log
server_audit_file_rotate_size=100000000
server_audit_file_rotations=9
########################
sql-mode=""
ignore-db-dir=lost+found
===
Previously MySQL-5.6 was used, then it was removed and Percona Server was installed up over existing database files.
{{}}