Reduce frequency of S3 header GET requests during agent health checks
General
Escalation
General
Escalation
Description
i'm using pbm-agent with PITR enabled to backup mongodb to backblaze s3 compatible service.
they have a daily free tier limit of 2500 calls to the API. which runs out in a matter of a couple of hours due to pbm-agent calling the "get s3 object header" endpoint every 5 seconds.
is this by design or is it a bug? seems like a bug to me because it does this every 5 seconds even when there are not PITR slices to upload (because nothing changed in the db).
if this is not a bug how do others handle the egress bills when pbm-agent calls s3 api like this in an excessive manner?
please check the log file attached with more information.
Environment
os version: ubuntu 20.10 pbm version: percona-backup-mongodb_1.4.1-1.bionic_amd64.deb
Attachments
1
Smart Checklist
Activity
Show:
Jira Bot July 30, 2021 at 10:56 AM
To: Former user CC:
Hi, I'm jira-bot, Percona's Jira automation tool. I've detected that someone from Percona has made an edit to the Summary field of an issue that you reported.
I'm not sentient (yet) so I'm not sure whether the person fixed a typo, changed a few words, or completely rewrote the text. In any case, it is Percona Engineering's intention to make the Summary and Description of an issue as accurate as possible so that we're fixing the actual problem you're encountering, and to avoid misunderstandings about symptoms and causes.
If the current Summary does not accurately reflect the problem you are reporting, or if you feel the change was otherwise inappropriate in some way, please add a new comment explaining things and we'll address it as soon as we can.
This message will be added only once per issue, regardless of how many times the Summary is edited.
message-code:summary-edited
Akira Kurogane March 8, 2021 at 7:05 AM
Fair point. We didn't think of a requests per day limit.
i'm using pbm-agent with PITR enabled to backup mongodb to backblaze s3 compatible service.
they have a daily free tier limit of 2500 calls to the API. which runs out in a matter of a couple of hours due to pbm-agent calling the "get s3 object header" endpoint every 5 seconds.
is this by design or is it a bug? seems like a bug to me because it does this every 5 seconds even when there are not PITR slices to upload (because nothing changed in the db).
if this is not a bug how do others handle the egress bills when pbm-agent calls s3 api like this in an excessive manner?
please check the log file attached with more information.