finalizers delete-s3-backup with GCS protection will cause an OOM
General
Escalation
General
Escalation
Description
Everytime the operator tires to delete a backup from an S3 bucket that has a retention policy configured. The controller retries the delete action multiple times (as evident from the logs), but each time it gets the same error, indicating the retention policy is still in effect.
The pod's memory usage might increase due to constant retries, caching, and other overheads associated with managing the delete operation and handling failures. This could potentially lead to an OOM (Out of Memory) issue if the pod's memory limits are too tight and the retries are frequent.
backup.yaml
backup-secret-s3.yaml
When trying to delete:
Memory increasing:
And here are the logs (also attached):
Environment
None
AFFECTED CS IDs
CS0040984
Attachments
2
Activity
Slava Sarzhan December 28, 2023 at 12:09 PM
Hi, the issue was fixed and will be available in the next PXCO release. The bug was in go code. We did not change the finalizer logic.
Everytime the operator tires to delete a backup from an S3 bucket that has a retention policy configured. The controller retries the delete action multiple times (as evident from the logs), but each time it gets the same error, indicating the retention policy is still in effect.
The pod's memory usage might increase due to constant retries, caching, and other overheads associated with managing the delete operation and handling failures. This could potentially lead to an OOM (Out of Memory) issue if the pod's memory limits are too tight and the retries are frequent.
backup.yaml
backup-secret-s3.yaml
When trying to delete:
Memory increasing:
And here are the logs (also attached):