Backup and Restore Service  ·  View 05 of 26  ·  2 · People and journeys

Journey — On-call SRE Recovers From a Three-Week-Old Corruption

The recovery the platform is really built for: a bad migration found weeks late, with no known start time and good data written since.

Editable source SVG draw.io All views
On-call SRE orders incident Goal — Recover orders to the last clean second without losing today Trigger — Finance finds totals wrong since a migration weeks ago Done when — Clean rows merged back, measured inside the RTO 1 · Paged 2 · Find clean point ◆ moment of truth 3 · Restore side by side 4 · Reconcile 5 · Cut over ◆ moment of truth 6 · Review What they do Opens incident Names the invariant Requests PITR copy Diffs clean vs live Gets second approval Reads drill timings System behind it Restore console Clean-point search Restore orchestrator Partial restore Target snapshot Evidence store How it feels In control Uneasy Stressed Where it hurts Start time unknown 35 days to search Log replay is slow Good writes since then Fear of a second loss What the platform gives Runbook in the page 8-way parallel bisect Hourly base bounds replay Table-level export Reversible in 7 days RTO counted, drill or not Journey — On-call SRE Recovers From a Three-Week-Old Corruption v 1.0 · owner Platform Architecture · date 2026-09

The trough, and what answers it

  • Finding the clean point is where this goes wrong. Guessing wastes a four-hour restore per guess. Clean-point search restores eight points at once and narrows down in about seven rounds (view 16).
  • Cutover is the second worst moment, because an in-place restore can destroy the only copy of today's good writes. Side by side is the default, and in-place needs a second approver and a snapshot taken first (view 17).

Decisions

  • The recovery restores side by side and copies back only the damaged rows. Rolling a whole database back three weeks is almost never what the business wants.
  • Hourly block-incremental base copies for Tier 1 keep WAL replay under an hour. Replay, not transfer, is usually what breaks an RTO, and it never appears on a storage vendor's throughput sheet.

Stated limits

  • The 60-minute RTO holds for recovery points in the last 7 days, where hourly bases are kept. From day 8 to day 35, only daily bases remain and the stated RTO is 3 hours. Older points have no PITR, only weekly and monthly copies.
  • Reconciling rows the application wrote after the corruption belongs to the service team. The platform provides the clean copy and a table-level diff, not business rules.