Evidence ledger
25 sources
Checked 11 Sep 2026
Evidence ledger
One row per claim in A backup is a claim, a restore is the proof: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic: how production teams prove a backup restores, read from the public repository record. All URLs fetched 2026-09-11 in this session. This session's network egress was restricted to git hosting (gitlab.com, github.com raw) and cloud.google.com; engineering-blog, paper and talk hosts were unreachable, so the ledger is built from repository and operator records plus one reachable vendor postmortem. That constraint is stated in the guide.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | GitLab | Postmortem of database outage of January 31 (source markdown, www-gitlab-com @ 2017 ref) | postmortem | 2017-02-10 | 2026-09-11 | https://gitlab.com/gitlab-com/www-gitlab-com/-/raw/30d10754fb46472e/source/posts/2017-02-10-postmortem-of-database-outage-of-january-31.html.md | Five backup/replication mechanisms existed; none produced a usable restore; recovery used an informal snapshot | "the process of both finding and using backups failed completely"; "When we went to look for the pg_dump backups we found out they were not there. The S3 bucket was empty" |
| 2 | GitLab | Same postmortem | postmortem | 2017-02-10 | 2026-09-11 | (same as #1) | The backup job failed for a year-class reason: version mismatch between tool and database | "pg_dump 9.2, while our database is running PostgreSQL 9.6 … pg_dump producing an error, terminating the backup procedure" |
| 3 | GitLab | Same postmortem | postmortem | 2017-02-10 | 2026-09-11 | (same as #1) | Failure notifications failed silently too | "these notifications are sent by email … DMARC was not enabled for the cronjob emails" |
| 4 | GitLab | Same postmortem | postmortem | 2017-02-10 | 2026-09-11 | (same as #1) | Restore duration was set by the cheapest disk in the chain | "Copying the data from the staging to the production host took around 18 hours … throttled to a really low number (around 60Mbps)"; staging on cheap storage "primarily done to save costs" |
| 5 | GitLab | Same postmortem | postmortem | 2017-02-10 | 2026-09-11 | (same as #1) | Impact figures | "roughly 5,000 projects, 5,000 comments and 700 new user accounts" lost from a ~6-hour window; ~300 GB removed |
| 6 | GitLab | Issue #1102: Automated testing of recovering PostgreSQL database backups | adr | 2017-02-01 | 2026-09-11 | https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/1102 | The automated-restore-test design was written the day after the outage, using the production tools | "This restore procedure should use the same underlying tools as a production engineer might use … removing the need for engineers to do this manually" |
| 7 | GitLab | Issue #1168: Backup system and replication is not working | postmortem | 2017-02-11 | 2026-09-11 | https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/1168 | One day after the postmortem published, backups were broken again, same WAL mechanism | "our backups are simply not working"; "requested WAL segment 0000000200000A74000000A1 has already been removed" |
| 8 | GitLab | Issue #1161: [Meta] Disaster recovery for everything that is not the database | adr | 2017-02-10 | 2026-09-11 | https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/1161 | Snapshot-existence and restorability named as different properties | "one thing is having snapshots, a different thing is having a backup that we can recover from" |
| 9 | GitLab | runbooks: disaster-recovery/README.md | source | live, checked 2026 | 2026-09-11 | https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/disaster-recovery/README.md | Restore validation runs as CI pipelines for both database and disk snapshots | "Validation of restores happen in CI pipelines for both the Postgresql database and disk snapshots" |
| 10 | GitLab | runbooks: disaster-recovery/gameday.md | source | live, checked 2026 | 2026-09-11 | https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/disaster-recovery/gameday.md | Weekly mock DR events; four-level confidence taxonomy in which "High" requires recovery tested in production | "Mock DR events are simulated almost every week"; High confidence = "We have tested recovery in production" |
| 11 | GitLab | runbooks: disaster-recovery/recovery-measurements.md | casestudy | 2024–2026 entries | 2026-09-11 | https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/disaster-recovery/recovery-measurements.md | Rehearsed restores are timed and logged per drill | Gitaly VM recreation 00:14–01:16 across 2024–2026 gamedays; zonal DR process 02:38 (2026-07-14, GSTG); "45 Production Gitaly VMs" in 00:39 (2024-10-21) |
| 12 | GitLab | runbooks: patroni/postgresql-backups-wale-walg.md | source | updated through ~2021+, checked 2026 | 2026-09-11 | https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/patroni/postgresql-backups-wale-walg.md | Backup scale/throughput figures; daily verification wired to the restore project | "As of September 2021, the daily backup process performed by WAL-G takes ~9 hours with Postgres cluster size ~14 TiB"; backup-fetch "used in 'gitlab-restore' for daily verification of backups"; WAL-G "gives 3-4x" faster basebackup restore than WAL-E |
| 13 | GitLab | Handbook: Backups of GitLab.com (controlled policy document) | adr | live, checked 2026 | 2026-09-11 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/gitlab-com/policies/backup/_index.md | Current backup shape: hourly full + WAL archival, 14-day retention, GCS soft delete 7 days; snapshot restore validation by random sampling | "A full backup is taken every hour, with continuous transaction log archival"; Git repo restore validation "Conducted by randomly sampling disks and restoring recent snapshots" |
| 14 | GitLab | Handbook: Disaster Recovery of GitLab.com (controlled policy document) | adr | live, checked 2026 | 2026-09-11 | https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/main/content/handbook/engineering/gitlab-com/policies/disaster-recovery/_index.md | Daily PITR restore test with integrity queries; hourly random Gitaly snapshot restore; object storage explicitly exempted | "performs a point-in-time recovery (PITR) restore into a new instance and verifies data integrity by running queries"; "Hourly restoration testing … using a randomly selected Gitaly disk"; "Automated restore validation is not required for Object Storage due to its inherent protections" |
| 15 | GitLab | Production incident #4056: Our database backups are failing to verify | postmortem | 2021-03-24 | 2026-09-11 | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/4056 | A failing verification pipeline is treated as a declared production incident | Title; incident declared 20:18 UTC, customer impact recorded as "DB Backup Verification" |
| 16 | GitLab | Production incident #3338: database failover triggered by GCP snapshots | postmortem | 2021-01-16 | 2026-09-11 | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/3338 | The backup process is itself production load; snapshots on all nodes concurrently caused a primary failover | "snapshots were initiated on all database servers concurrently, under the mistaken impression that the snapshot process was a transparent background transaction" |
| 17 | GitLab | Production incident #22543: gitlab-restore/postgres-gprd snitch missing | postmortem | 2026-07-20 | 2026-09-11 | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22543 | The 2017 version-drift failure class recurred in 2026 but was caught by the dead-man's-snitch as a severity-4 non-event | "The restore pipelines were still configured for pg16 after postgres was upgraded to pg17"; "There does not appear to be any customer impact" |
| 18 | GitLab | MR !231770: Remove --unlink-first and --recursive-unlink from tar extract (closed, unmerged) | source | 2026-04-16 | 2026-09-11 | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/231770 | An OS security patch broke the restore path's tar invocation; restore code rots against its environment | "These flags cause restore to fail on RHEL/EL9+ systems where a patched GNU tar (RHSA-2026:0067) makes unlink(\".\") return EINVAL" |
| 19 | GitLab | MR !229743: Add Barman backup option for large database (closed, unmerged draft) | source | 2026-03-31 | 2026-09-11 | https://gitlab.com/gitlab-org/gitlab/-/merge_requests/229743 | Named gap: pg_dump-based backup stops being practical above ~100 GB; physical/WAL backup is the flip | "for Omnibus-bundled PostgreSQL databases larger than 100 GB, the existing documentation recommends using cloud-managed database services"; Barman "significantly faster than logical backups via pg_dump" |
| 20 | GitLab | Docs: Restore GitLab (restore_gitlab.md) | vendor | live, checked 2026 | 2026-09-11 | https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/backup_restore/restore_gitlab.md | The restore contract is version-exact | "You can only restore a backup to exactly the same version and type (CE or EE)" |
| 21 | GitLab | gitlab-restore group (project index) | source | group created 2019-01-08 | 2026-09-11 | https://gitlab.com/groups/gitlab-com/gl-infra/gitlab-restore | The restore-verification practice is organised as a group of CI projects, one per datastore | Group description "Backup restore procedures for GitLab."; projects: "Restore procedures for our main database in gprd", "Create and verify disk snapshots in CI for Production", "Ansible playbooks that restore and verify a Gitaly GCP snapshot" |
| 22 | Google Cloud | Details of Google Cloud GCVE incident | postmortem | 2024-05-24 | 2026-09-11 | https://cloud.google.com/blog/products/infrastructure/details-of-google-cloud-gcve-incident | A blank provisioning parameter set a customer's private cloud to auto-delete after a fixed term; backups outside the deleted scope were what recovery stood on | "one input parameter was left blank … the system assigned a then unknown default fixed 1 year term value"; "Data backups that were stored in Google Cloud Storage in the same region were not impacted"; third-party backup software "instrumental in aiding the rapid restoration" |
| 23 | restic | doc/design.rst (repository design document) | adr | in-repo, checked 2026 | 2026-09-11 | https://github.com/restic/restic/blob/master/doc/design.rst | The tool's threat model: trusted host, untrusted storage; integrity must be verifiable against the storage | "The host system a backup is created on is trusted. This is the most basic requirement, and it is essential for creating trustworthy backups." |
| 24 | restic | doc/045_working_with_repos.rst (checking integrity) | vendor | in-repo, checked 2026 | 2026-09-11 | https://github.com/restic/restic/blob/master/doc/045_working_with_repos.rst | Default check is structural only; full data verification costs a full download, so the documented alternative is sampling | "Since --read-data has to download all pack files in the repository, beware that it might incur higher bandwidth costs"; --read-data-subset=n/t and random x% sampling |
| 25 | PostgreSQL | doc/src/sgml/backup.sgml (Continuous Archiving docs source) | vendor | in-repo, checked 2026 | 2026-09-11 | https://github.com/postgres/postgres/blob/master/doc/src/sgml/backup.sgml | Physical backups are all-or-nothing and archiving must be monitored | "file system backups only work for complete backup and restoration of an entire database cluster"; WAL archiving "must be used with caution: if WAL archiving is not monitored correctly" |
| 26 | etcd | op-guide/recovery.md (docs source, v3.5) | vendor | in-repo, checked 2026 | 2026-09-11 | https://github.com/etcd-io/website/blob/main/content/en/docs/v3.5/op-guide/recovery.md | Restoring the data does not restore the system: clients observe revisions going backwards; Kubernetes informer caches go inconsistent | "existing clients may perceive the revision going back"; "Restoring to an older revision may not correctly refresh the caches, causing unpredictable and inconsistent behavior in the controllers"; copying member/snap/db "might lose data that has not been written yet, but is included in the wal" |
| 27 | GitHub | backup-utils README | source | in-repo, checked 2026 | 2026-09-11 | https://github.com/github/backup-utils/blob/master/README.md | GHES backup architecture: incremental snapshots with retention; the tool itself is being replaced | "Incremental backup of Git repository data. Only changes since the last snapshot are transferred"; "backup-service becomes GA in GHES 3.20 … backup-utils will be deprecated in GHES 3.22" |
| 28 | Velero (VMware) | design/restore-progress.md | adr | in-repo, checked 2026 | 2026-09-11 | https://github.com/vmware-tanzu/velero/blob/main/design/restore-progress.md | Restores were long opaque: a phase, no progress; the design records the gap | "Velero Restore, on the other hand, only shows one of the phases (InProgress, Completed, PartiallyFailed, Failed) of the ongoing restore" |
| 29 | Kubernetes | KEP-177: CSI Volume Snapshot | adr | in-repo, checked 2026 | 2026-09-11 | https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/177-volume-snapshot/README.md | The platform snapshot primitive assumes quiescing is the application's job | "The application is usually quiesced before taking a snapshot to ensure consistency" |
Notes on absence
- No papers and no talks are cited. The FAST/USENIX backup-systems literature and conference talks exist but their hosts were unreachable from this session's restricted network, as were all engineering-blog hosts (Atlassian's 2022 post-incident review, Roblox's 2021 postmortem, Cloudflare's November 2023 DR account). The guide says so and confines itself to what was fetched and verified. Claims those unreachable sources would support are not made.
- gitlab-restore pipeline repositories (bootstrap.sh etc.) mirror from ops.gitlab.net and their file contents are not anonymously readable; the pipelines' existence, purpose and cadence are cited from the runbooks, the controlled policy documents, and the group/project descriptions, which are public.