HA_ERR_ROW_IS_REFERENCED - FK constraint fail, node consistency compromised - after PXC updates only
General
Escalation
General
Escalation
Description
Environment
x86_64 EL7, on the day of the latest incident just updated to latest 7.9.2009.
is caused by
is duplicated by
Smart Checklist
Activity
Show:
Ville Ojamo March 18, 2022 at 2:38 PM
We have now started to have this exact same problem again.
Happened on 2022-03-11 and 2022-03-18, both on Percona-XtraDB-Cluster-57-5.7.35-31.53.1.el7.x86_64 (2022-03-11) and Percona-XtraDB-Cluster-57-5.7.36-31.55.1.el7.x86_64 (2022-03-18).
Venkatesh Prasad January 6, 2021 at 12:34 PM
This is a duplicate of
Ville Ojamo November 18, 2020 at 6:50 AM
The formatting on the pastes turned out bad especially for the log. Here is another attempt at it:
Duplicate
Details
Details
Assignee
Venkatesh PrasadReporter
Ville OjamoAffects versions
Priority
MediumSmart Checklist
Open Smart Checklist
Smart Checklist
Open Smart Checklist
Created November 18, 2020 at 6:40 AM
Updated March 6, 2024 at 9:20 PM
Resolved January 6, 2021 at 12:36 PM
This error has happened so far two times in 2020 and very curiously both times the problem has happened immediately after Percona XtraDB Cluster update:
On July 7, update PXC from 5.7.25-31.35.1.el7.x86_64 to 5.7.30-31.43.1.el7.x86_64
On November 17, update PXC from 5.7.30-31.43.1.el7.x86_64 to 5.7.31-31.45.3.el7.x86_64
The statement that causes the issue is run every night on schedule. But on July 7 it caused 2 nodes to exit with "node consistency compromised, aborting..." and on November 17 it caused 1 node to exit with same error message.
In 2020 PXC was updated only 2 times and both times the cluster encountered this error the first time the scheduled job was run after the update.
Logs from the November 17 incident:
The tables in question are tables for app "OTRS" version 6 and they do have primary keys:
Out-of-order commit is disabled, parallel apply was enabled with 24 threads but has since been changed to disabled in order to see if it helps:
Binary log from one of the servers that staid up shows, in this order:
deletes from table communication_log_object_entry
deletes from table communication_log_obj_lookup
deletes from table communication_log_object
deletes from table communication_log