A backup is a claim, a restore is the proof

GitLab lost six hours of production data in 2017 because five backup mechanisms produced zero working restores; the same failure class returned in July 2026 as a severity-4 non-event because by then a CI pipeline restored the database every day and a dead-man switch noticed the silence. This guide reconstructs the architecture between those two dates from GitLab's postmortems, incident tracker, runbooks and controlled policy documents, Google's GCVE deletion report, and the design records of restic, Velero, PostgreSQL, etcd and GitHub's backup tooling. A reader leaves able to say which of their backups has an exercised restore path, what cadence and sampling depth each datastore deserves, and why the alert must fire on silence rather than on failure.

What you get out of it

• Redundant backup mechanisms are one mechanism until each has an independently exercised restore path; GitLab had five on paper and zero in practice on the night it mattered.
• Alert on the verifier's silence, not the backup's failure: 2017's failure emails died in DMARC, while the 2020s dead-man snitch caught every drift incident on the record.
• Verification depth is a budgeted quantity: full read-back downloads the whole repository (restic's documented warning), so production practice converges on daily full restores for the critical database and random sampling for bulk fleets.
• The failure domain includes your account and provider: the GCVE deletion arrived through Google's own control plane, and the copy that mattered was held by a different custodian.
• Restore time is set by the cheapest link and measured only by drills: 18 hours over throttled 60 Mbps disks in 2017, versus gameday logs that time every rehearsed recovery since 2024.

Scope

Why this, now. Google's 2024 GCVE deletion and GitLab's 2026 stream of snitch-missing incidents put restore verification on the public record at both extremes, the provider deleting the cloud and the verifier catching drift weekly, so the practice is auditable in public for the first time.

What it does not cover. Ransomware-specific immutability products, tape and media management, legal-deletion pipelines, application-level export formats, and multi-region active-active designs; engineering-blog and conference-paper accounts (Atlassian 2022, Roblox 2021, Cloudflare 2023, USENIX FAST) were unreachable from this session's network and are deliberately not cited.