Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Server crash on select from a specific table / possible corruption

Description

Hi,

Today an old innodb table (name=order_online_order) with 300-400 rows started to crash suddenly percona server 8.0.30-22. No DDL query executed for weeks.

Logs show that random select queries to the table cause this and create a stack trace. Here is the list of things i tried after it started to happen:

  • random "select x,y,z from table" queries often CRASHED.

  • "select count from table" query always CRASHED.

  • "optimize, analyze table" queries always CRASHED.

  • "select 1 from table" always WORKED.

  • "select primary_key_column from table limit 1" always WORKED.

  • creating another table, and running "insert into new table from old table" WORKED.

  • drop table WORKED.

After copying data to a duplicate table schema, i dropped the original one successfully and for now it works ok. Seems like sth is corrupted, but not the data, because i could fetch them in to new table.

Error log is attached. At 14.01.2023 we created this server from a xtrabackup backup and it works ok, until we encountered this issue. You can see that there is no log between 14.01.2023 and 23.01.2023, and the crash first happens around 13:00 and constantly repeats with different queries and sometimes different error codes.

Environment

Attached.

Attachments

3

Activity

Aaditya Dubey December 10, 2023 at 8:36 AM

Hi ,

Closing the report, no activity for a long!

Aaditya Dubey January 30, 2023 at 8:55 AM

Hi ,

Sure, Thank you!

Arda Beyazoglu January 30, 2023 at 8:44 AM

Hi,

Okay then. If we encounter again, I will try to prepare an environment where you can check actively.

Aaditya Dubey January 30, 2023 at 8:37 AM

Hi ,

Thank you for the updates.
We have seen issue due to "ALTER ALGORITHM=INSTANT", However until we don't have reproducible steps we are unable to find real culprit behind it.

Please go through below URLs where "ALTER ALGORITHM=INSTANT" is being discussed.

https://forums.mysql.com/read.php?22,704532,704685

https://jira.percona.com/browse/PS-8292

Arda Beyazoglu January 25, 2023 at 3:44 PM

Hi,

The query was cut like that in error.log so i can't share the full query. We will try to reproduce the same situation this week and i will share more details if we experience the same thing again.

In any case, everything works in the newly created table for now too, so i am not suprised you couldnt reproduce it.

A missing information from the report: We usually add new columns using INSTANT algorithm. Since recreated table doesn't have the issue (at least for now), is it possible instant ddl corrupted table at some point ?

 

Done

Details

Assignee

Reporter

Needs QA

Yes

Affects versions

Priority

Smart Checklist

Created January 23, 2023 at 3:41 PM
Updated December 10, 2023 at 8:37 AM
Resolved December 10, 2023 at 8:37 AM