Issues
- PITR restore hung with duplicate key errorK8SPXC-1420Resolved issue: K8SPXC-1420ege.gunes
- .metadata.ownerReferences contains duplicate entriesK8SPXC-1392Resolved issue: K8SPXC-1392natalia.marukovich
- PXC backup and restore job pods do not bear the classic app.kubernetes.io/* labels used by other componentsK8SPXC-1378Resolved issue: K8SPXC-1378Pavel Tankov
- Allow running several backup restores for different cluster in the parallelK8SPXC-1319ege.gunes
- Auto tune PBM configurationK8SPSMDB-1097natalia.marukovich
- Operator can start two PBM backup operations if backup object is updatedK8SPSMDB-1088Resolved issue: K8SPSMDB-1088Pavel Tankov
- Reconciler error: invalid syntaxK8SPS-380natalia.marukovich
- [investigation] Find a way to improve our deploy files generationK8SPS-363George Kechagias
- pgbackrest debug logs is hard to readK8SPG-732George Kechagias
- Support custom images in major upgradeK8SPG-598Resolved issue: K8SPG-598dmitriy.kostiuk
- Run DROP EXTENSION on extension deletionK8SPG-594Resolved issue: K8SPG-594dmitriy.kostiuk
- Improve MakefileK8SPG-589inel.pandzic
- Operator must stop WAL watcher if namespace or cluster does not existK8SPG-588Resolved issue: K8SPG-588Julio Pasinatto
- Make it harder to corrupt backup repo by promoting a standby clusterK8SPG-558ege.gunes
- wrong log level message when data source doesn't existK8SPG-532natalia.marukovich
- cluster stuck in init state if pgbackrest secret doesn't existK8SPG-499Resolved issue: K8SPG-499Eleonora Zinchenko
- Cluster status must be ready only if all statefulsets are up to dateK8SPG-454Resolved issue: K8SPG-454Julio Pasinatto
- `Labels for all the k8s objects created by OperatorK8SPG-329natalia.marukovich
18 of 18
PITR restore hung with duplicate key error
Done
General
Escalation
General
Escalation
Description
Environment
None
Attachments
3
Created July 16, 2024 at 1:23 AM
Updated December 19, 2024 at 8:11 PM
Resolved December 2, 2024 at 8:57 AM
Activity
Eleonora ZinchenkoNovember 21, 2024 at 10:27 AM
Hi,
Verified. Proxies are scaled down to 0 during pitr restore, pitr finishes successfully. Agreed with that we will add to tests check that proxies are scaled down during pitr restore so moving task back to in progress.
Slava SarzhanNovember 8, 2024 at 10:03 AM
Hi , thank you for task. We have improved this PITR restoration behavior. During PITR operator will not start (scale down) proxy pods. It was added into 1.16.0. You can use main image if you want to test it.
It seems that HAProxy is exposed at the time when PITR restore pod is running and if they have conflicting transactions it would lead to PITR pod stuck on duplicate key error.
Test:
Create database and table
Create one script that inserts data and execute
Create one script that shows latest inserts and execute
Current Inserts at 3:30PM PHT:
Scheduled Full Backup at 3:30PM PHT:
Current inserts at 3:33PM PHT:
Stop writes and truncate table at 3:35PM PHT
Content of table at 3:35PM PHT
Apply PITR up to 3:34PM PHT
Start insert script again. Insert script cannot connect now.
Restore process has begun. However, strangely, HAProxy has been allowed external access to the MySQL at the same time when PITR restore job is running at the very bottom of this output:
From the select script you can see that full backup has been restored but not PITR. Also, new data has been inserted on the table:
From the logs, you can see that the restore pod is still running but is not able to apply PITR due to duplicate key error:
On Everest side, the status is still in “Restoring” state