delete backup finalizer fails with PBM v2 with old storage
General
Escalation
General
Escalation
Description
We have a scenario in our backup test where we run backup on different storages and then at the end delete them, but it seems we can delete only backup which was running on last storage defined. For some steps to reproduce try to use latest PBM v2, define aws and gcp storages with "prefix" folder defined and run with sharding (I don't think all these are needed, but currently it's easiest to reproduce).
STR: 1. configure aws storage and run backup1 on aws 2. configure gcp storage and run backup2 on gcp 3. delete backup2 (should be fine) 4. try to delete backup1 and it should fail with something like: "Waiting for delete to be done ..Error: deleting: delete files from storage: delete "2023-01-10T15:40:19Z.pbm.json": no such file" or similar. (but the file exist in the bucket and in our subdirectory defined by prefix) Same scenario seems to work fine with PBM v1 for us.
Here's output of some commands: manual delete
backups list:
pbm status
operator log error
pbmConfig content
I can easily reproduce this so if you wish we can have some call to check it out while running.
I'm aware that the subject of this ticket will probably change completely, but it requires more checks to find the root of the issue.
Environment
None
Activity
Tomislav Plavcic January 11, 2023 at 11:02 AM
Hi !
Thanks for the comment, any idea why I was not able to reproduce this with PBM v1?
If I read correctly what you wrote there was no change about this in v2, but I can only reproduce this with v2.
andrew.pogrebnoi January 11, 2023 at 10:14 AM
HI ,
After changing the storage, PBM shouldn't see any artefacts from the previous one. So if you have bcp1 made on AWS, pbm status shouldn't show it after switching to GCP. PBM works with only one storage at a time and has no access to the previous one. pbm cli does automatically run resync on any config storage changes. So I rekon this situation is specific to operators as you use "internal API" and probably don't trigger resync after the storage change. So pbm still holds metadata of the backups from previous storage whilst doesn't have physical access to it (it rather assumes these files are present on current storage).
We have a scenario in our backup test where we run backup on different storages and then at the end delete them, but it seems we can delete only backup which was running on last storage defined.
For some steps to reproduce try to use latest PBM v2, define aws and gcp storages with "prefix" folder defined and run with sharding (I don't think all these are needed, but currently it's easiest to reproduce).
STR:
1. configure aws storage and run backup1 on aws
2. configure gcp storage and run backup2 on gcp
3. delete backup2 (should be fine)
4. try to delete backup1 and it should fail with something like: "Waiting for delete to be done ..Error: deleting: delete files from storage: delete "2023-01-10T15:40:19Z.pbm.json": no such file" or similar.
(but the file exist in the bucket and in our subdirectory defined by prefix)
Same scenario seems to work fine with PBM v1 for us.
Here's output of some commands:
manual delete
backups list:
pbm status
operator log error
pbmConfig content
I can easily reproduce this so if you wish we can have some call to check it out while running.
I'm aware that the subject of this ticket will probably change completely, but it requires more checks to find the root of the issue.