Atlassian uses cookies to improve your browsing experience, perform analytics and research, and conduct advertising. Accept all cookies to indicate that you agree to our use of cookies on your device. Atlassian cookies and tracking notice, (opens new window)
After the last backup at 4:30 pm, delete 2 backup schedules and change the time of the remaining 1 schedule, so that there are no further backups taken by pg.
Then do restores many times using backup, PITR like
For the last restore, the backup taken at 4:15 pm was chosen, which is before the latest backup taken at 4:30 pm
The restore gets stuck and the pg operator starts crashing with the error
Tested with Everest v1.1.0-rc3 version and did not observe this issue.
Oksana Grishchenko
July 22, 2024 at 10:44 AM
The PG limitations from the https://perconadev.atlassian.net/browse/EVEREST-1213 epic are implemented, so the described problems are not the case anymore bc there is no option to create two schedules with the same storage anymore. Moving the ticket to Ready for QA in case it needs an additional validation.
Even though we haven’t properly fixed this issue, we have taken preemptive actions that should forbid the use cases that would trigger this issue. See https://perconadev.atlassian.net/browse/EVEREST-1169 for more details about the applied restrictions.
Tested with Everest 1.0.0 rc1.
Create a pg db with 2 nodes, version 16.1. Enable 3 schedules for hourly at different times with PITR.
Add data in the database and let many backups complete.
postgresql-iyf-repo2-full-45ggp postgresql-iyf Succeeded 5h24m 5h25m postgresql-iyf-repo2-full-7gpmg postgresql-iyf Succeeded 4h24m 4h25m postgresql-iyf-repo2-full-cjmks postgresql-iyf Succeeded 3h24m 3h25m postgresql-iyf-repo2-full-mcxkd postgresql-iyf Succeeded 6h24m 6h25m postgresql-iyf-repo2-full-rhmr7 postgresql-iyf Succeeded 9h 9h postgresql-iyf-repo2-full-xbcdn postgresql-iyf Succeeded 8h 8h postgresql-iyf-repo2-full-xxtp7 postgresql-iyf Succeeded 7h24m 7h25m postgresql-iyf-repo3-full-bz97v postgresql-iyf Succeeded 7h9m 7h10m postgresql-iyf-repo3-full-g2q98 postgresql-iyf Succeeded 8h 8h postgresql-iyf-repo3-full-ld84f postgresql-iyf Succeeded 5h9m 5h10m postgresql-iyf-repo3-full-lf5gz postgresql-iyf Succeeded 4h9m 4h10m postgresql-iyf-repo3-full-mz8tk postgresql-iyf Succeeded 3h9m 3h10m postgresql-iyf-repo3-full-r8l2v postgresql-iyf Succeeded 9h 9h postgresql-iyf-repo3-full-xbbtt postgresql-iyf Succeeded 6h9m 6h10m postgresql-iyf-repo4-full-8nbjv postgresql-iyf Succeeded 3h54m 3h55m postgresql-iyf-repo4-full-k5xs5 postgresql-iyf Succeeded 5h54m 5h55m postgresql-iyf-repo4-full-m4289 postgresql-iyf Succeeded 4h54m 4h55m postgresql-iyf-repo4-full-mrs67 postgresql-iyf Succeeded 6h54m 6h55m postgresql-iyf-repo4-full-zr498 postgresql-iyf Succeeded 7h54m 7h55m postgresql-iyf-repo4-full-zrk4x postgresql-iyf Succeeded 8h 8hAfter the last backup at
4:30 pm, delete 2 backup schedules and change the time of the remaining 1 schedule, so that there are no further backups taken by pg.Then do restores many times using backup, PITR like
For the last restore, the backup taken at
4:15 pmwas chosen, which is before the latest backup taken at4:30 pmThe restore gets stuck and the pg operator starts crashing with the error
2024-06-18T12:42:33.764Z INFO Waiting for restore to start {"controller": "perconapgrestore", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGRestore", "PerconaPGRestore": {"name":"restore-c6m","namespace":"n1"}, "namespace": "n1", "name": "restore-c6m", "reconcileID": "fe71f89c-4f09-4da8-ba16-b6a41a08386d", "request": {"name":"restore-c6m","namespace":"n1"}} 2024-06-18T12:42:33.814Z INFO Starting workers {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "worker count": 1} 2024-06-18T12:42:33.831Z INFO Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference {"controller": "perconapgbackup", "controllerGroup": "pgv2.percona.com", "controllerKind": "PerconaPGBackup", "PerconaPGBackup": {"name":"postgresql-iyf-repo3-full-cqckh","namespace":"n1"}, "namespace": "n1", "name": "postgresql-iyf-repo3-full-cqckh", "reconcileID": "853c7d5f-fefb-4ef6-a5c8-abee7522ecce"} panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1685f49] goroutine 534 [running]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:116 +0x1e5 panic({0x1a233e0?, 0x2ddbe70?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/percona/percona-postgresql-operator/percona/controller/pgbackup.(*PGBackupReconciler).Reconcile(0xc0007c0560, {0x1fcc410, 0xc000595350}, {{{0xc00101b34e, 0x2}, {0xc0009df8c0, 0x1f}}}) /go/src/github.com/percona/percona-postgresql-operator/percona/controller/pgbackup/controller.go:103 +0x1009 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1fcf718?, {0x1fcc410?, 0xc000595350?}, {{{0xc00101b34e?, 0xb?}, {0xc0009df8c0?, 0x0?}}}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119 +0xb7 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0007aea00, {0x1fcc448, 0xc00078b5e0}, {0x1abf5c0?, 0xc000318760?}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316 +0x3cc sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0007aea00, {0x1fcc448, 0xc00078b5e0}) /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266 +0x1c9 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227 +0x79 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 145 /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:223 +0x565Attaching complete logs for the pg operator.