In January 2017 GitLab lost roughly six hours of database data after an engineer deleted a directory on the wrong host during replication troubleshooting, and then found that several backup and replication mechanisms had silently not been working. What signal would have revealed the broken backups beforehand, and why did nothing report them?

The trigger

An engineer working late on replication problems ran a destructive command against the wrong host. That part is ordinary: every organisation has this failure available to it at all times, and the interesting question is never how to prevent every misdirected command.

The interesting part is what happened next. Of the several recovery mechanisms believed to exist, the ones that mattered did not work, and this was discovered during the incident.

Why nothing reported it

Each mechanism failed in a way that produced no error anyone was watching for.

• A dump that failed because of a version mismatch between the tool and the server, writing a small or empty file, while the wrapping job still finished.
• Notification of failures going to an address nobody read, which converts an alert into an archive.
• Snapshots that existed but were not enabled for the host in question, so a control believed to be universal was not.
• A process that completed successfully and produced an artefact nobody had ever read back.

The common structure: every check was on the production side of the operation, and none was on the recovery side. A backup system monitored by whether the backup job ran is monitoring the wrong event. The event that matters is a successful restore, and it was never performed.

Why detection lagged until the worst moment

Backup failures are discovered at restore time by default, and restore time is, by definition, the worst moment available. There is no natural feedback: nothing in normal operation ever reads a backup, so the entire subsystem can rot indefinitely while its dashboards stay green.

The structural fix versus the tempting local fix

The tempting fixes are more confirmation prompts, red shell prompts on production hosts and a rule about working alone at night. All are worth doing, all address the trigger, and none of them would have preserved the data.

The structural fix is to make recovery a routine, exercised operation:

1. Automated restore to a scratch environment on a schedule, with validation queries on the restored copy. This turns every one of the failures above into a next-day alert.
2. Alert on time since last verified restore, with a threshold shorter than the interval in which a change could break the pipeline. Prefer 7 days as the default threshold; 30 days is the outer limit for anything important. The cost is a scratch environment and some storage, against a loss measured in hours of customer data.
3. Independent copies with different mechanisms and different credentials, so one broken tool or one compromised account cannot take them all.
4. The restore procedure executed by someone who did not write it, from the documentation, under time pressure, at least once a year. That exercise finds the missing key, the undocumented step and the permission nobody has.

The general lesson

An unverified backup is a belief, not a control, and the gap between the two is invisible until the day it matters. The measure of a backup system is the age and provenance of the last successful restore, and nothing else. GitLab published a detailed public account in 2017, which is why this incident is still the most useful teaching case in the area.

When this is the wrong emphasis

If the data can be rebuilt deterministically from an upstream source, the rebuild is the recovery path and it is the thing to exercise. Testing backups you would never actually use is theatre, and the effort belongs on the path you would genuinely take.