Percona backup: unable to do PITR restore when doing bulk insert
Description
Environment
Entries in pbmPITRChunks
{{rs0:PRIMARY> db.pbmPITRChunks.find()
{ "_id" : ObjectId("604dfff66c3acc761ba2e45a"), "rs" : "rs0", "fname" : "pbmPitr/rs0/20210314/20210314121204-3.20210314122209-2.oplog.snappy", "compression" : "s2", "start_ts" : Timestamp(1615723924, 3), "end_ts" : Timestamp(1615724529, 2) }
{ "_id" : ObjectId("604e00c86c3acc761ba2e45d"), "rs" : "rs0", "fname" : "pbmPitr/rs0/20210314/20210314122209-2.20210314122539-3.oplog.snappy", "compression" : "s2", "start_ts" : Timestamp(1615724529, 2), "end_ts" : Timestamp(1615724739, 3) }}}
pbm list
{{Backup snapshots:
2021-03-14T12:11:42Z [complete: 2021-03-14T12:12:04]
PITR <off>:
2021-03-14T12:12:04 - 2021-03-14T12:25:39}}
and when I try to restore it with pbm restore I am getting error
{{ *pbm restore --time="2021-03-14T12:12:04"*
..Error: define last oplog slice: get: mongo: no documents in result}}
pbm agent logs
2021-03-14T19:57:09.000+0545 I got command pitrestore [name: 2021-03-14T14:12:08.9699Z, point-in-time ts: 1615723924] <ts: 1615731128>
2021-03-14T19:57:09.000+0545 I got epoch {1615724729 2}
2021-03-14T19:57:09.000+0545 I [pitr/2021-03-14T12:12:04Z] recovery started
2021-03-14T19:57:10.000+0545 E [pitr/2021-03-14T12:12:04Z] restore: define last oplog slice: get: mongo: no documents in result
Above restore produces this query in mongo and this clearly does not take counter of timestamp into account.
"op" : "query",
"ns" : "admin.pbmPITRChunks",
"command" : {
"find" : "pbmPITRChunks",
"filter" : {
"rs" : "rs0",
"start_ts" : {
"$lte" : Timestamp(1615723924, 0)
},
"end_ts" : {
"$gte" : Timestamp(1615723924, 0)
}
},
"limit" : 1,
"singleBatch" : true,
"readConcern" : {
"level" : "majority"
},
"lsid" : {
"id" : UUID("5c31c699-58b7-4835-aa58-56300a56ce25")
},
"$clusterTime" : {
"clusterTime" : Timestamp(1615737655, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : 0
}
},
"$db" : "admin",
"$readPreference" : {
"mode" : "primary"
}
},
Smart Checklist
Activity
Akira Kurogane March 22, 2021 at 5:17 AM
Thanks the update Sameer. I'll close this Jira case.
Sameer Kattel March 18, 2021 at 5:46 PM
Hi Andrii,
thanks for the response. yep I got it. yep I can work with what you have suggested
andrew.pogrebnoi March 16, 2021 at 4:47 PM
Hi
Thanks for reporting. Yes, I see. When PBM tries to define a chunk containing a specified time it looks for chunk with `start_time <= time && start_time >= time`. And since only time is specified (without the "counter" part) it assumes counter = 0. It makes issues when the specified time is equal to the beginning of the available timeframe (like in your case). Because it looks lest's say for `"start_ts": {$lte: Timestamp(1613985290, 0)}` where is first available time is `Timestamp(1613985290, 5)`.
But again this is the issue only for the first second in PITR timeframes.
If you'll try to restore to the next second - it will work (`pbm restore --time="2021-03-14T12:12:05` in your case). But to restore to the 2021-03-14T12:12:04 you should use snapshot (`pbm restore 2021-03-14T12:11:42Z` in your case) because this is the end time of the snapshot (see "complete at" column in `pbm list`) e.g. restore would be up to that time.
We should probably change `pbm list` so it won't reflect this overlapping second in "PITR chunks" section.
Sameer Kattel March 16, 2021 at 3:21 AM
Cannot seem to edit this ticket , I guess I was supposed to put environment content into description.
Also initially I asked this question in https://forums.percona.com/t/percona-backup-unable-to-do-pitr-restore/9649 froum and I am pretty sure this is a bug.
PITR restore does not take counter part of timestamp into account. So unable to do pitr restore as agent never finds entry in
pbmPITRChunks collections because of this.