Add global delete-backup finalizer that removes all the backups on cluster deletion

Description

In our other operators, we have a delete-backup finalizer under backup CR, meaning we can delete the individual backup.


In PG we can’t do that, the only thing we can delete is all the backups from all the repos or backups for a particular repo.

The question is, do we want/need some finalizer to handle backup deletion? The only one for us that could make sense is to have a finalizer under cluster CR, like delete-backups so when you delete your cluster you will delete backups if you want.

Environment

None

Attachments

1
  • 27 Jan 2025, 08:47 PM

Activity

Show:

Julio Pasinatto January 30, 2025 at 7:00 PM

Hi, the issue that mentioned is now fixed. delete-backupsfinalizer works correctly, just deleting the backups of the same cluster they belong to.

Eleonora Zinchenko January 27, 2025 at 8:47 PM
Edited

Hi,

delete-backup  finalizer deletes all the  pg-backup objects (of all the clusters) present in namespace.

STR:

  1. Deploy cluster1 with delete-backup  finalizer commented out

  2. Deploy cluster2 with delete-backup  finalizer present.

  1. Delete cluster2. The pg-backup objects of both cluster1 and cluster2 are deleted ... Operator log is flooded with ERROR   WALWatcher      get latest backup Errors for cluster1 

In logs that I am attaching I have at first deleted cluster1 and its pg-backup object was kept. Then deleted cluster2 - both backup objects were deleted. But provided STR was also checked - all the present in namespace pg-backup objects were deleted.

Additionally, what is also strange, is that based on logs after the cluster is created and backup succeeded - there are messages about running & removing finilizer for cluster that did not have it enabled & there are reconsiler errors related to update of backup objects:

2025-01-27T20:03:39.730Z INFO Backup succeeded {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "5af64ea7-f452-47fe-80eb-9482db2d0f8d", "request": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}} 2025-01-27T20:03:39.730Z INFO Running finalizer {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "5af64ea7-f452-47fe-80eb-9482db2d0f8d", "name": "percona.com/delete-backup"} 2025-01-27T20:03:40.241Z INFO Removing finalizer {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "5af64ea7-f452-47fe-80eb-9482db2d0f8d", "name": "percona.com/delete-backup"} 2025-01-27T20:03:40.782Z INFO Got latest restorable timestamp {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "ccc1bbff-8864-4260-978e-973a36004f65", "request": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "timestamp": "2025-01-27 20:02:38.412922 +0000 UTC"} 2025-01-27T20:03:40.796Z ERROR Reconciler error {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "ccc1bbff-8864-4260-978e-973a36004f65", "error": "update PGBackup status: Operation cannot be fulfilled on perconapgbackups.pgv2.percona.com \"cluster1-backup-6lrn-fbs2c\": the object has been modified; please apply your changes to the latest version and try again", "errorVerbose": "Operation cannot be fulfilled on perconapgbackups.pgv2.percona.com \"cluster1-backup-6lrn-fbs2c\": the object has been modified; please apply your changes to the latest version and try again\nupdate PGBackup status\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgbackup.(*PGBackupReconciler).Reconcile\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgbackup/controller.go:262\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:319\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:279\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:240\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:279 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:240 2025-01-27T20:03:41.534Z INFO Got latest restorable timestamp {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "8f081dd2-d559-4c07-b220-14bfc9734c9a", "request": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "timestamp": "2025-01-27 20:02:38.412922 +0000 UTC"} 2025-01-27T20:03:42.292Z INFO Got latest restorable timestamp {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "0d57410a-5738-4b27-bb66-1a9518e40da6", "request": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "timestamp": "2025-01-27 20:02:38.412922 +0000 UTC"} 2025-01-27T20:03:42.300Z ERROR Reconciler error {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"cluster1-backup-6lrn-fbs2c","namespace":"backfin"}, "namespace": "backfin", "name": "cluster1-backup-6lrn-fbs2c", "reconcileID": "0d57410a-5738-4b27-bb66-1a9518e40da6", "error": "update PGBackup status: Operation cannot be fulfilled on perconapgbackups.pgv2.percona.com \"cluster1-backup-6lrn-fbs2c\": the object has been modified; please apply your changes to the latest version and try again", "errorVerbose": "Operation cannot be fulfilled on perconapgbackups.pgv2.percona.com \"cluster1-backup-6lrn-fbs2c\": the object has been modified; please apply your changes to the latest version and try again\nupdate PGBackup status\ngithub.com/percona/percona-postgresql-operator/percona/controller/pgbackup.(*PGBackupReconciler).Reconcile\n\t/go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgbackup/controller.go:262\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:319\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:279\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:240\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:279 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:240

Attaching operator log.

Sergey Pronin May 30, 2024 at 7:17 AM

Lets add the following:

metadata: finalizers: - delete-backups ...

It will be commented in default CRs and helm chart.

In our docs we will state that it is not recommended for production use.

Assignee

dmitriy.kostiuk

Reporter

Labels

Needs QA

Yes

Needs Doc

Yes

Story Points

Fix versions

Priority

Created May 29, 2024 at 7:34 AM
Updated January 30, 2025 at 7:00 PM

Flag notifications