Innodb crash upgrading from 5.7 to 8
Description
Environment
Ubuntu 18.04, on an i3.metal in AWS using ZFS
This node is a replica, not the master.
Attachments
- 07 Jan 2019, 08:58 PM
- 07 Jan 2019, 08:58 PM
duplicates
Smart Checklist
Activity
Jericho Rivera January 8, 2019 at 9:33 AM
Marked as Duplicate of https://jira.percona.com/browse/PS-5245
Jericho Rivera January 8, 2019 at 4:40 AM
@Ashley Penney
This is a duplicate of https://perconadev.atlassian.net/browse/PS-5425#icft=PS-5425. We will update ticket status accordingly, I can't seem to find a way to properly set this bug report as duplicate of another bug report.
In the meantime, please subscribe to the other bug report to receive updates.
Ashley Penney January 7, 2019 at 8:59 PM
root@ip-10-102-53-11:/evq/mysql/log# dpkg -l | grep percona
ii percona-release 1.0-1.bionic all Package to install Percona gpg key and APT repos
ii percona-server-client 8.0.13-3-2.bionic amd64 Percona Server database client binaries
ii percona-server-common 8.0.13-3-2.bionic amd64 Percona Server database common files (e.g. /etc/mysql/my.cnf)
rc percona-server-common-5.7 5.7.23-25-1.bionic amd64 Percona Server database common files (e.g. /etc/mysql/my.cnf)
iF percona-server-server 8.0.13-3-2.bionic amd64 Percona Server database server binaries
rc percona-server-server-5.7 5.7.23-25-1.bionic amd64 Percona Server database server binaries
ii percona-toolkit 3.0.12-1.bionic amd64 Advanced MySQL and system command-line tools
ii percona-xtrabackup-24 2.4.12-1.bionic amd64 Open source backup tool for InnoDB and XtraDB
I attached a clean error.log and the full /etc/mysql directory (which is a mess, as this thing has been lugged around for a while now). I meddled with a bunch of the settings based on the error I was getting, specifically the key_buffer_size, read_buffer_size, max_threads, etc. We had these set higher in production, but I kept stepping them down as they were called out as potential issues.
No other storage engine than InnoDB as far as I'm aware!
Jericho Rivera January 4, 2019 at 2:31 AM
Please share the full mysql error log as well as the my.cnf of the affected node. We'd also appreciate it if you can share the exact Percona Server 8.0 packages installed on the server. Do you happen to use other storage engines aside from InnoDB?
Hi,
I attempted to upgrade our 35TB database from Percona 5.7 up to the recently released 8.0 release. (I also tried the slightly newer one in ps80-testing). Upon startup I get:
2019-01-02T20:02:15.188880Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000
100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000
2019-01-02T20:04:32.723622Z 2 [ERROR] [MY-013183] [InnoDB] Assertion failure: dict0dict.cc:1230:table2 == NULL thread 140668865402624
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
20:04:32 UTC - mysqld got signal 6 ;
From the backtrace:
Thread pointer: 0x556255d608f0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ff005c0cda0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace(unsigned char*, unsigned long)+0x3d) [0x556252a5908d]
/usr/sbin/mysqld(handle_fatal_signal+0x433) [0x556251d8a373]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7ff01a8f6890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7ff018adae97]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7ff018adc801]
/usr/sbin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x2d0) [0x5562
52d9c710]
/usr/sbin/mysqld(dict_table_add_to_cache(dict_table_t*, unsigned long, mem_block_info_t*)+0x320)
[0x556252e46390]
/usr/sbin/mysqld(+0x20bf30d) [0x556252e7930d]
/usr/sbin/mysqld(dict_load_table(char const*, bool, dict_err_ignore_t)+0x78f) [0x556252e79fff]
/usr/sbin/mysqld(dict_table_open_on_name(char const*, unsigned long, unsigned long, dict_err_ign
ore_t)+0x180) [0x556252e4e150]
/usr/sbin/mysqld(dd_upgrade_table(THD*, char const*, char const*, dd::Table*, TABLE*)+0xf1) [0x5
56252e96761]
/usr/sbin/mysqld(+0x1c5bdd8) [0x556252a15dd8]
/usr/sbin/mysqld(dd::upgrade_57::migrate_all_frm_to_dd(THD*, char const*, bool)+0x588) [0x556252
a176d8]
/usr/sbin/mysqld(dd::upgrade_57::fill_dd_and_finalize(THD*)+0x1c2) [0x556252a07dc2]
/usr/sbin/mysqld(+0x107c5b6) [0x556251e365b6]
/usr/sbin/mysqld(+0x1d4a2af) [0x556252b042af]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7ff01a8eb6db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7ff018bbd88f]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 2
Status: NOT_KILLED
I've tried tweaking a bunch of settings but I'm not really sure what else to try next!