On 31 January 2017 GitLab lost about six hours of database data after an engineer removed the wrong directory during replication troubleshooting. The company had five backup and replication mechanisms. None of them produced a usable recent restore. What does that say about the difference between a control being designed and a control operating?

The situation they were in

GitLab published the incident in detail while it was still running, which is why it is teachable. A tired engineer working late on replication problems ran a directory removal against the primary rather than the intended replica and stopped it after a second or two. Roughly 300 GB was already gone.

Then came the part that matters. Several protection mechanisms existed on paper — logical backups, snapshots, replication, a copy to object storage. Each had failed quietly: one produced near-empty files because of a version mismatch that only printed a warning, others were not configured where people believed they were. The recovery in the end came from a snapshot taken by chance hours earlier for an unrelated reason.

Around 6 hours of data was lost: on the order of 5,000 projects, 5,000 comments and 700 accounts.

What the distinction actually means

A control has two separate properties, and an audit tests them separately. Design effectiveness asks whether the control, if it ran as described, would prevent or detect the thing it exists for. Operating effectiveness asks whether it actually ran, over the whole period, and produced the result.

GitLab's backups were well designed. Not one of them was operating. The difference is not paperwork: it is the difference between believing you can recover and being able to.

The reason this class of failure is so common is that a backup job's success signal is that it exited zero, which tests the job and not the backup. The only test of a backup is a restore.

What it cost, and the general lesson

The public cost was the lost data and a day of degraded service. The lesson that transfers is narrower than "test your backups":

• A control with no output that anyone consumes will stop working and nobody will notice. Attach a consumer: a monthly restore into a scratch environment, with the row count compared against production.
• Warnings are not signals. The version mismatch printed a warning nobody read, which is the normal fate of warnings.
• Verify the property, not the process. "Backup job succeeded" is a process signal. "A restore from last night reached 99.9% of production's row count in 45 minutes" is a property signal.

Where copying this is a mistake

The tempting conclusion is to add more backup mechanisms. Five mechanisms were the problem's context, not its solution: each additional one dilutes attention and makes it more likely that each is someone else's responsibility. One mechanism with a proven, timed, scheduled restore beats five unverified ones, and it is cheaper.

Common weak answers

• "They needed better change management." The deletion was the trigger; the outage's length was set by recovery, and recovery is what failed.
• "Automate the backup checks." Automate the restore. A check that asserts a file exists tests a file.