How production teams prove they can get their data back, reconstructed from the public operational record: GitLab's postmortems, runbooks, policies and restore pipelines from 2017 to 2026, Google's account of deleting a customer's private cloud, and the design documents of the backup tools themselves.
The problem, stated without naming a product: when the data a system runs on is destroyed, does a second copy exist that can actually be brought back, completely, within a bounded time, from a failure domain the disaster does not share, and how do you know that before the day it matters?
On 31 January 2017 an engineer deleted roughly 300 GB from GitLab.com's primary database. The postmortem's sentence about what happened next is the founding document of this whole topic: "the process of both finding and using backups failed completely." The pg_dump job had been failing for months because the tool was version 9.2 and the database was 9.6, so the S3 bucket was empty. The failure emails had been silently rejected for want of a DMARC signature. Azure disk snapshots were not enabled for the database hosts. Replication was the thing that had just broken. The recovery ran from a staging LVM snapshot an engineer happened to have taken six hours earlier, copied back over disks that a cost decision had throttled to about 60 Mbps, for 18 hours.
On 20 July 2026, the same organisation hit the same failure class again: after an upgrade to PostgreSQL 17, the restore pipeline was still configured for PostgreSQL 16. The difference is the whole subject of this guide. In 2017 the version mismatch was discovered at 23:00 on the worst night of the company's life, because the only thing that ever exercised the backups was a disaster. In 2026 it was discovered by a Dead Man's Snitch on a CI pipeline that restores the production database every day, filed at severity 4, and fixed by changing two variables. Nine years of work did not eliminate the failure class. It demoted it.
This guide reconstructs how teams that have been burned close that chain: what the verification layer looks like, which decisions shape it, what has broken in production, and the numbers a design review can plan against. It is built almost entirely from repository records: postmortems and incident issues, runbooks, controlled policy documents, merge requests and tool design docs, because that is where this practice actually lives, in public, with dates on it.
This dig was researched from a network position that could reach git hosting (gitlab.com, github.com) and cloud.google.com but not engineering-blog or conference hosts. Atlassian's 2022 post-incident review, Roblox's 2021 postmortem, Cloudflare's November 2023 disaster-recovery account and the USENIX FAST backup-systems literature all exist and are relevant; they were unreachable this session, are not cited, and no claim below leans on them. The corpus is therefore deep on GitLab, which operates its production infrastructure in the open, plus Google's GCVE incident report and five tools' own records. Treat the GitLab material as one organisation's account, corroborated where the tool documentation independently agrees.
Deliberately out of scope: ransomware-specific immutability products, tape and media management, legal-deletion pipelines, application-level export formats, and multi-region active-active designs where nothing is ever restored because everything is always live. Those are adjacent problems with their own records.
The common shape of a backup system that can prove itself, assembled from GitLab's policies and runbooks, Google's incident report, and the design documents of restic, Velero, PostgreSQL and etcd. Every box is attributable.
Strip away the products and four structural commitments recur.
Two backup kinds per datastore, because they fail differently. GitLab's current policy pairs a continuous WAL archive with hourly disk snapshots for the same database, and hourly block snapshots for Git data. The PostgreSQL documentation is blunt about why one kind is not enough: file-system-level backups "only work for complete backup and restoration of an entire database cluster", while WAL archiving gives point-in-time precision but "must be used with caution: if WAL archiving is not monitored correctly", the chain silently loses its value. The 2017 outage is the degenerate case: five mechanisms that were really one mechanism, because none had an independent, exercised restore path.
The copy lives outside the failure domain, and the failure domain includes your account. The 2024 Google Cloud incident is the sharpest published lesson here. A blank parameter in an internal provisioning tool gave UniSuper's private cloud "a then unknown default fixed 1 year term value", and at the end of the term the platform deleted the deployment. Google's own report notes that backups stored in Google Cloud Storage survived, and that the customer's "third party backup software" was "instrumental in aiding the rapid restoration". The transferable reading: deletion arrived through the provider's own control plane, so the copy that mattered most was the one a different organisation held. GitLab's policy encodes the same idea one level down, with GCS soft delete at 7 days and object versioning as a firebreak between "deleted" and "gone".
A machine restores on a schedule and a human is paged only when it goes quiet. This is the part most teams do not build. GitLab's controlled DR policy commits to a daily CI pipeline that "performs a point-in-time recovery (PITR) restore into a new instance and verifies data integrity by running queries", and to hourly verification that takes "a randomly selected Gitaly disk", restores it to a new disk, and checks for recent commits. The design dates to 1 February 2017, the day after the outage, in issue #1102: the restore test "should use the same underlying tools as a production engineer might use", exercised automatically, "removing the need for engineers to do this manually". The alerting on top is deliberately inverted: the pipeline pings a dead-man snitch on success, and the alert fires on silence. The 2017 outage had alert-on-failure, and the failure emails died in DMARC.
The restore path is code, and it rots like code. The 2026 incident record shows what the verification layer actually catches in a steady state: a PostgreSQL major upgrade the pipeline had not followed (July 2026), missing scheduled GCP snapshots (June 2026, March 2026), snitch gaps after refactors (May and June 2026). A closed merge request from April 2026 shows the same rot arriving from below: a Red Hat security patch to GNU tar made GitLab's restore invocation fail on EL9 systems, because the restore code passed flags whose behaviour the patch changed. None of these were data loss. All of them would have been, on the day, if the first time the path ran was the disaster.
GitLab's restore practice is a group of CI projects, one per thing that must come back: the main database, the subscriptions database, Gitaly disks, Ops and Staging snapshots. Each is a pipeline whose job is a restore.
Recorded at: gitlab-restore group, since 2019
restic's documentation warns that a full data check "has to download all pack files in the repository", with the bandwidth bill that implies, and documents random and n-of-t sampling as the affordable alternative. GitLab samples Gitaly disks hourly rather than restoring the fleet.
Recorded at: restic docs, GitLab backup policy
GitLab's gameday runbook defines four confidence levels per service, and the top one is unreachable by analysis: "High confidence" requires "We have tested recovery in production". Mock DR events run almost weekly; each drill's wall-clock numbers are committed to the runbook.
Recorded at: gameday.md, recovery-measurements.md
"One thing is having snapshots, a different thing is having a backup that we can recover from." GitLab infrastructure meta-issue #1161, "Disaster recovery for everything that is not the database", 10 February 2017
Each of these is a fork with a published rationale on at least one side, and a condition that flips it.
check validates structure; reading the data is a separate, costed
flag. A checksum proves the bytes; it says nothing about version drift, tooling,
credentials or wall-clock time.| Decision | Chosen | Rejected | Because | Flips when | Evidence |
|---|---|---|---|---|---|
| Proof of restorability | Scheduled real restores | Artifact checksums alone | Restores exercise tooling, versions, credentials and time, not just bytes | Cadence and depth scale down with blast radius | GitLab DR policy |
| Verifier alerting | Dead-man switch on success pings | Email on failure | The failure channel failed silently in 2017 (DMARC) | Does not flip | Incident #22543 |
| Verification depth | Random sampling (hourly disk, n-of-t packs) | Full read-back every cycle | Full read-back downloads the whole repository; bandwidth and time | Small repositories, or after any corruption sighting | restic docs |
| Database backup kind | Physical base + WAL (PITR) | pg_dump as the disaster path | Speed and point-in-time precision at multi-TiB scale | Below tens of GB, logical wins on portability | WAL-G runbook, MR !229743 |
| Where the last copy lives | Outside the account and provider scope | All copies under one control plane | The GCVE deletion arrived through the provider's own tooling | Never fully; cost decides how much lives outside | Google, 2024 |
| Object storage restore tests | Exempt, in writing | Testing everything uniformly | Versioning plus soft delete cover the deletion vector the tests would catch | The store's own account or region is the thing at risk | GitLab backup policy |
The exemption row deserves a sentence, because it is the only place the two best sources in this corpus disagree in spirit. GitLab exempts object storage from restore testing on the strength of versioning and soft delete. Google's GCVE report shows a deletion vector that ran above those protections: the subscription itself was deleted by the platform. Both are right in their own threat model. The reader's job is to say, in writing, which threat model their exemption assumes; the flip condition is the moment that assumption stops holding.
Six published incidents, grouped into the four classes this corpus keeps producing: drift severs the restore path, the notification channel fails silently, the backup shares the primary's failure domain, and the backup process is itself production load.
A fourth class hides behind all of these: restoring the data is not restoring the system. etcd's documentation is the cleanest statement in the corpus. After a snapshot restore, "existing clients may perceive the revision going back", and for Kubernetes specifically, "restoring to an older revision may not correctly refresh the caches, causing unpredictable and inconsistent behavior in the controllers". GitLab met the same class in 2017: the recovered snapshot had webhooks deliberately stripped (it was a staging copy), so webhooks were surgically re-imported from a second snapshot, and every database sequence was advanced by 100,000 so restored IDs could not collide with IDs handed out before the outage. A restore plan that ends at "data directory present" has not planned the part where the system re-enters a world that moved on.
Everything quantitative this corpus yields, dated and sourced. The pattern worth carrying: backup duration, restore duration and drill duration are three different numbers, and only the third one is measured with humans in the loop.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Data removed by the accident | ~300 GB | GitLab | rm -rf on the primary; ~4.5 GB remained | 2017-01 | postmortem |
| Data loss window | ~6 h | GitLab | ~5,000 projects, ~5,000 comments, ~700 accounts | 2017-01 | postmortem |
| Restore copy time | ~18 h | GitLab | staging disks throttled to ~60 Mbps, a cost choice | 2017-01 | postmortem |
| Cluster size | ~14 TiB | GitLab | main GitLab.com PostgreSQL cluster | 2021-09 | runbook |
| Daily full backup duration | ~9 h | GitLab | WAL-G backup-push, run on a secondary | 2021-09 | runbook |
| Backup push throughput | 0.5-1 TiB/h | GitLab | "very IO-intensive"; a capacity line item | 2021 | runbook |
| WAL-G vs WAL-E restore | 3-4× | GitLab | basebackup fetch phase, n1-standard-16 | 2020 | runbook |
| Backup cadence, database | hourly + WAL | GitLab | full backup every hour, continuous log archival | 2026 | policy |
| Retention / soft delete | 14 d / 7 d | GitLab | backups 14 days; GCS soft delete 7 days | 2026 | policy |
| Restore test cadence, database | daily | GitLab | PITR into a fresh instance, integrity queries | 2026 | DR policy |
| Restore test cadence, Git data | hourly | GitLab | random Gitaly disk, recent-commit check | 2026 | DR policy |
| Gameday: recreate 45 prod Gitaly VMs | 39 min | GitLab | production drill, plus ~11 min bootstrap | 2024-10 | measurements |
| Gameday: zonal DR process | 2 h 38 m | GitLab | staging, end to end including MRs and checks | 2026-07 | measurements |
| Deletion timer nobody knew about | 1 year | Google GCVE | blank parameter defaulted a fixed term | 2023-2024 | |
| GCVE recovery effort | days, 24x7 | Google / UniSuper | third-party backups called "instrumental" | 2024-05 |
All figures above are operator-reported, not independently measured, and the GitLab throughput numbers date from 2021 against hardware that has since changed. Two quantities this corpus does not contain, because the reachable record does not publish them: the money cost of running a daily restore pipeline, and any cross-organisation distribution of restore times. Where a number was needed and none exists, this page says so rather than estimating.
Every source behind this page, graded. All links were fetched and read on 2026-09-11. No papers and no talks appear: their hosts were unreachable from this session's network, which is stated rather than papered over.
The canonical backup-failure postmortem, read here from its markdown source in the website repository at a 2017 commit. Five mechanisms enumerated, each with the specific reason it produced nothing restorable.
Written the day after the outage: a weekly host restores the latest backup with the production tools, checks that users, projects and issues contain data, and alarms on failure. The design the next decade implemented and tightened to daily.
Eleven days after the outage, one day after the postmortem published: the WAL segment the secondary needed was already gone, and "our backups are simply not working". The rawest artifact in this corpus.
The widening of scope, twelve numbered workstreams, and the sentence quoted at the top of section 02 distinguishing having snapshots from having a recoverable backup.
The operator-facing map: backups tolerant of zonal and regional loss via multi-region object storage, with restore validation running "in CI pipelines for both the Postgresql database and disk snapshots".
Mock DR events "almost every week", explicit roles, and a four-level confidence taxonomy per service in which High confidence requires recovery tested in production, not in a production-like environment.
A running log of drill timings: Gitaly VM recreation between 14 and 76 minutes across eight dated entries, a 2 h 38 m zonal DR process, notes on the drill where node boot-up needed debugging.
The operational anatomy: daily base backup from a Consul-elected secondary, WAL shipped continuously to one bucket per environment, ~9 h for ~14 TiB, restore automated in the gitlab-restore CI project, delayed replica for fast PITR.
The audited policy: hourly full database backups with continuous log archival, 14-day retention, GCS soft delete at 7 days, snapshot restore validation by random sampling, and object storage explicitly exempted with the reason written down.
Daily PITR restore testing with data-integrity queries for the main and subscriptions databases; hourly random-disk Gitaly snapshot restores verified by recent commits; quarterly gamedays validating RTO and RPO against recorded measurements.
A declared production incident whose customer impact field reads "DB Backup Verification". The verification pipeline failing is treated with the same machinery as user-facing outages: declared, timelined, given corrective actions.
Concurrent snapshots on all database nodes, begun "under the mistaken impression that the snapshot process was a transparent background transaction", cost the primary its Patroni lock inside four minutes.
The surprise at the top of this page: post-upgrade version drift, 2017's signature failure, caught by the restore pipeline's dead-man switch as a severity-4 item with no customer impact. Sibling incidents in May and June 2026 show the switch working repeatedly.
A distribution's security patch to GNU tar broke GitLab's restore invocation on EL9+. The change removes flags the restore never needed; closed without merge, the argument preserved.
A closed draft that names the gap in official guidance: bundled databases over roughly 100 GB have outgrown pg_dump-based backups, and the documented answer, cloud-managed databases, is unavailable to many self-managed customers.
The restore contract, stated flatly: "You can only restore a backup to exactly the same version and type (CE or EE)" of GitLab it was created on. The sentence that makes version drift a standing threat rather than an edge case.
The practice as an org chart artifact: "Backup restore procedures for GitLab", with CI projects for the gprd database, the subscriptions database, Gitaly snapshot verification, and snapshot verification for Production, Staging and Ops.
Google's own account of deleting UniSuper's private cloud: the blank parameter, the unknown one-year term, the 24x7 multi-day recovery, and the sentence crediting the customer's third-party backup software as instrumental.
The repository format's threat model in one line: "The host system a backup is created on is trusted", the storage is not, and everything in the format follows from making integrity checkable against untrusted storage.
The default check is structural; verifying actual data needs --read-data, which downloads every pack file, and the docs say so with a bandwidth warning. Sampling via --read-data-subset, by fraction or percentage, is the documented compromise.
The constraints that shape every serious deployment: file-system backups restore the whole cluster or nothing; WAL archiving must be monitored or it silently loses its guarantee; corrupted WAL ends recovery early at an implicit target.
Restoring is documented as a semantic event, not a copy: clients can observe the revision going backwards, Kubernetes informers may hold caches the restore invalidates, and a snapshot taken by copying the db file can miss unflushed WAL.
GitHub Enterprise Server's backup tooling: incremental snapshots transferring only changes, configurable retention, parallel backup and restore, and a deprecation notice as backup moves into the product ("backup-service becomes GA in GHES 3.20").
The design opens by recording the asymmetry it fixes: backups had real-time progress, restores exposed only a phase. The operator watching a long restore had no way to distinguish progress from a hang.
The platform's snapshot primitive, with the consistency contract stated in passing: the application "is usually quiesced before taking a snapshot", which is to say the platform snapshot is crash-consistent and application consistency is your hook to write.
Seven rungs from an evening's toy to the shape GitLab runs. The line from toy to real is crossed at rung 4, where a machine starts proving the restore without you.
Run a database with sample data, take a logical dump, destroy the database, restore into a fresh instance and diff row counts per table.
Done when: the diff is empty and you wrote down how long each step took. Teaches: the restore path has more steps than the backup path, and the timings are not symmetric.
Enable WAL archiving to object storage (any S3-compatible store), take a base backup, write for an hour, then restore to a timestamp ten minutes before a deliberate bad write.
Done when: the bad write is absent and everything before it is present. Teaches: RPO is a property of the archive chain, and the docs' warning about monitoring that chain becomes concrete the first time a segment is missing.
Upgrade the database one major version and re-run yesterday's restore procedure unchanged. Then downgrade the client tools and try again.
Done when: you have seen a green backup fail to restore for a reason that no checksum would catch. Teaches: the failure class behind GitLab's 2017 and 2026 incidents, on your own machine.
A scheduled pipeline restores the latest backup into a fresh instance, runs three integrity queries, and pings a snitch endpoint on success. Alert when the ping is absent, not when the job reports failure.
Done when: stopping the backup job pages you within one schedule interval, with nobody watching. Teaches: alert-on-silence, which is the load-bearing inversion in the whole architecture.
For file or snapshot data, restore one randomly chosen artifact per cycle and check freshness (GitLab checks a restored Gitaly disk for recent commits). For a restic repository, schedule read-data-subset across the week.
Done when: the sampling schedule covers the corpus in a bounded window and the verification cost appears in your bill. Teaches: verification depth as a budgeted quantity, not a virtue.
Replicate the backup bucket to a different account with different credentials, or a different provider. Then run a drill assuming your primary account is hostile: restore using only the second copy.
Done when: the restore succeeds from credentials your primary admin cannot revoke. Teaches: the GCVE lesson, that replication inside one administrative scope is one copy.
Simulate losing a zone or the primary datastore, execute the runbook end to end with a colleague as reviewer, record every timing, and grade each service on GitLab's four-level confidence scale.
Done when: the measurements are committed next to the runbook and at least one service moved up a confidence level. Teaches: RTO as a measured, dated number, and how much of it is humans coordinating rather than bytes moving.
The queries that found this material. The first group works on any GitLab-hosted operator record; the third generalises to any organisation that runs its infrastructure in the open.
https://gitlab.com/api/v4/projects/7444821/issues?labels=incident&search=backuphttps://gitlab.com/api/v4/projects/<id>/issues?search=outage&created_after=2017-01-25&created_before=2017-03-01https://gitlab.com/api/v4/groups?search=gitlab-restorehttps://gitlab.com/api/v4/projects/<id>/merge_requests?state=closed&search=backup/repository/commits?path=source/posts&since=2017-02-09&until=2017-02-12/repository/files/<urlencoded-path>/raw?ref=<old-sha>git clone --depth 1 --filter=blob:none --sparse <repo> && git sparse-checkout set design docsite:gitlab.com runbooks backup restore testing"dead man's snitch" OR "snitch missing" backup restore incident"gameday" OR "game day" "disaster recovery" runbook measurements"restore" "we found" backup "was empty" postmortemrepo design/ directory: velero, restic doc/design.rst, keps/sig-storage"--read-data" OR "read-data-subset" site:github.com restic"exactly the same version" restore site:docs.gitlab.com