Add global delete-backup finalizer that removes all the backups on cluster deletion
Description
Environment
Attachments
- 27 Jan 2025, 08:47 PM
Activity
Julio Pasinatto January 30, 2025 at 7:00 PM
Hi, the issue that @Eleonora Zinchenko mentioned is now fixed. delete-backups
finalizer works correctly, just deleting the backups of the same cluster they belong to.
Eleonora Zinchenko January 27, 2025 at 8:47 PMEdited
Hi,
delete-backup
finalizer deletes all the pg-backup
objects (of all the clusters) present in namespace.
STR:
Deploy cluster1 with
delete-backup
finalizer commented outDeploy cluster2 with
delete-backup
finalizer present.
Delete cluster2. The
pg-backup
objects of bothcluster1
andcluster2
are deleted ... Operator log is flooded withERROR WALWatcher get latest backup
Errors for cluster1
In logs that I am attaching I have at first deleted cluster1 and its pg-backup
object was kept. Then deleted cluster2 - both backup objects were deleted. But provided STR was also checked - all the present in namespace pg-backup
objects were deleted.
Additionally, what is also strange, is that based on logs after the cluster is created and backup succeeded - there are messages about running & removing finilizer for cluster that did not have it enabled & there are reconsiler errors related to update of backup objects:
2025-01-27T20:03:39.730Z INFO Backup succeeded {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "5af64ea7-f452-47fe-80eb-9482db2d0f8d", "request": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}}
2025-01-27T20:03:39.730Z INFO Running finalizer {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "5af64ea7-f452-47fe-80eb-9482db2d0f8d", "name": "percona.com/delete-backup"}
2025-01-27T20:03:40.241Z INFO Removing finalizer {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "5af64ea7-f452-47fe-80eb-9482db2d0f8d", "name": "percona.com/delete-backup"}
2025-01-27T20:03:40.782Z INFO Got latest restorable timestamp {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "ccc1bbff-8864-4260-978e-973a36004f65", "request": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "timestamp": "2025-01-27 20:02:38.412922 +0000 UTC"}
2025-01-27T20:03:40.796Z ERROR Reconciler error {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "ccc1bbff-8864-4260-978e-973a36004f65", "error": "update PGBackup status: Operation cannot be fulfilled on perconapgbackups.pgv2.percona.com \"cluster1-backup-6lrn-fbs2c\": the object has been modified; please apply your changes to the latest version and try again", "errorVerbose": "Operation cannot be fulfilled on perconapgbackups.pgv2.percona.com \"cluster1-backup-6lrn-fbs2c\": the object has been modified; please apply your changes to the latest version and try again\nupdate PGBackup status\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgbackup.(*PGBackupReconciler).Reconcile\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgbackup/controller.go:262\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:319\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:279\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:240\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:279
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:240
2025-01-27T20:03:41.534Z INFO Got latest restorable timestamp {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "8f081dd2-d559-4c07-b220-14bfc9734c9a", "request": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "timestamp": "2025-01-27 20:02:38.412922 +0000 UTC"}
2025-01-27T20:03:42.292Z INFO Got latest restorable timestamp {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "0d57410a-5738-4b27-bb66-1a9518e40da6", "request": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "timestamp": "2025-01-27 20:02:38.412922 +0000 UTC"}
2025-01-27T20:03:42.300Z ERROR Reconciler error {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "0d57410a-5738-4b27-bb66-1a9518e40da6", "error": "update PGBackup status: Operation cannot be fulfilled on perconapgbackups.pgv2.percona.com \"cluster1-backup-6lrn-fbs2c\": the object has been modified; please apply your changes to the latest version and try again", "errorVerbose": "Operation cannot be fulfilled on perconapgbackups.pgv2.percona.com \"cluster1-backup-6lrn-fbs2c\": the object has been modified; please apply your changes to the latest version and try again\nupdate PGBackup status\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgbackup.(*PGBackupReconciler).Reconcile\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgbackup/controller.go:262\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:319\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:279\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:240\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:279
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:240
Attaching operator log.
Sergey Pronin May 30, 2024 at 7:17 AM
Lets add the following:
metadata:
finalizers:
- delete-backups
...
It will be commented in default CRs and helm chart.
In our docs we will state that it is not recommended for production use.
In our other operators, we have a delete-backup finalizer under backup CR, meaning we can delete the individual backup.
In PG we can’t do that, the only thing we can delete is all the backups from all the repos or backups for a particular repo.
The question is, do we want/need some finalizer to handle backup deletion? The only one for us that could make sense is to have a finalizer under cluster CR, like
delete-backups
so when you delete your cluster you will delete backups if you want.