Backup and Restore Service  ·  View 16 of 26  ·  5 · Runtime

Finding the Last Clean Recovery Point

How the platform finds the last clean second when the corruption's start time is unknown, which is the usual case.

Editable source SVG draw.io All views
On-call SRE Clean-point search Catalogue Isolated cluster Assertion runner Evidence store 1. invariant SQL · window 35 days 2. recovery ranges · log gaps 3. continuous range, 2 known gaps 4. restore 8 points, evenly spaced 5. 8 copies ready 6. clean up to p3, dirty from p4 7. narrow to [p3, p4] · repeat 8. round 7 · 1-second spacing 9. last clean 02:17:09 10. rounds · points · results 11. recovery point + confidence Finding the Last Clean Recovery Point When Nobody Knows When It Started Nine-way search over 3 million seconds converges in 7 rounds. Round cost is set by replay distance. v 1.0 · owner Backup Platform · date 2026-09

Decisions

  • The search is a nine-way bisection: restore 8 evenly spaced points, run the owner's invariant, and keep the interval where it turns from clean to dirty. 35 days is about 3 million seconds, and nine-way search needs 7 rounds to reach one second.
  • The invariant is a query the owner supplies, such as totals reconciling or no orphaned rows. Without one the platform can restore any second, but it cannot say which second is clean.
  • Search restores use the rehearsal slots and pause the Tier 2 rotation while they run. Tier 1 daily rehearsals keep running.

Numbers

  • Round cost depends on replay distance. Inside the 7-day hourly window, a round on a 1 TB store takes about 25 minutes, so the whole search is about 3 hours. With only daily bases, rounds take about an hour.
  • 8 parallel restores is the rehearsal-slot budget in the NFRs, not a coincidence.

Assumptions

  • The corruption is monotonic: once the invariant fails, it keeps failing. A corruption that was repaired and then happened again needs a person reading the round results, and the evidence record keeps every round for that reason.