Percona TokuDB Crashes causing file corruption

General

Escalation

General

Escalation

Description

TokuDB will occasionally crash with error:

mysqld: /mnt/workspace/percona-server-5.7-debian-binary-rocks-new/label_exp/min-buster-x64/test/percona-server-5.7-5.7.36-39/storage/tokudb/PerconaFT/portability/file.cc:137: void try_again_after_handling_write_error(int, size_t, ssize_t): Assertion `try_again' failed.

We haven't been able to identify the cause of this crash yet. We've enable log_syslog, but nothing we still get that line first and then a mysqld signal 6. No output prior to the assertion pertaining to mysql. I've included the output of our error log.

We were able to recover one server by copying the raw file from a replication server, but that's not a good resolution. I still have some time to try to reproduce this with some of the queries that we were running.

Transaction:

create table x like y

insert into table y select from table x where col != val

alter table rename x to temp_x

alter table y rename to x

drop table temp_x

Close Transaction

Environment

Debian 10

Attachments

Activity

Show:

Aaditya Dubey January 16, 2024 at 12:12 PM

We recommend Migrating the data to the MyRocks storage engine. To enable the plugins to migrate to another storage engine, set the tokudb_enabled and tokudb_backup_enabled options to TRUE in your my.cnf file and restart your server instance. Then, you can load the plugins.

Please note 5.7 is EOL and Starting with Percona Server for MySQL 8.0.28-19, the TokuDB storage engine is no longer supported. We have removed the storage engine from the installation packages and the storage engine is disabled in our binary builds.

Better upgrade to 8.0 and use MyRocks engine if required.

Thank you connecting Percona!

Aaditya Dubey January 14, 2024 at 4:24 PM

We couldn't repeat the issue that’s why this report is moved from confirmation to new, will try again and see if we can repeat it.

Jose Vargas December 13, 2023 at 3:42 PM

Aaditya,

I saw that this ticket has changed from Incomplete to Confirmation and now back to New. Why was this changed back to new?

Jose'

Aaditya Dubey May 12, 2023 at 9:08 AM

Hi ,

Thank you for the updates. Will check further and see if issue can be repeated from our end.

Jose Vargas May 4, 2023 at 1:23 PM

Aaditya,

On top of the that transaction existing, I was inserting data ranging from 500K to 100M records in order every 10 to 30 minutes. In a separate process. The transaction above is essentially our deduplication process for MySQL at the moment. I let that process run for a week or two with nothing else happening on the instance, and the db would crash 1 to 3 times in that period.

We are actually working on transitioning to InnoDB. We are unsure if the issue lies with the TokuDB engine or with Percona's variant of MySQL. We haven't tested this process using Percona with InnoDB, since the new stack we're working on migrating to is MariaDB's variant with InnoDB.

Jose'