On 31 January 2017 a GitLab engineer removed the wrong PostgreSQL data directory, and the recovery mechanisms they believed they had turned out not to work. Roughly six hours of data was lost. Which property had never been tested, and what does that say about non-functional testing generally?

The situation they were in

During replication troubleshooting late at night, a command intended for a broken secondary ran against the primary and removed roughly 300 GB before it was stopped. GitLab then worked through its recovery options in public, and the published postmortem records that several supposedly independent mechanisms were not usable.

The specifics are the lesson. The logical backup was failing silently because the client tooling was a different major version from the server, so it produced nothing useful and exited in a way nobody was watching. The notification emails about that failure were being rejected, so the silence was complete. Disk snapshots were not enabled for the database hosts. What saved them was a snapshot taken for an unrelated staging purpose about six hours earlier, which is why the data loss window was about six hours: roughly 5000 projects, 5000 comments and 700 new accounts.

The property that was never tested

That a backup can be restored. Every mechanism was verified by the existence of a job, not by the production of a usable database. A backup job's exit code is a claim. A restore is the test.

This generalises to a class of non-functional properties that cannot be verified by inspection, only by exercise: restore, regional failover, certificate and key rotation, scale-out under real load, queue drain after an outage, and the incident tooling itself. Each of them is a procedure whose first real execution is during the worst hour of the year unless you schedule it earlier.

What a strategy does with that

• Assert on the restored artefact, not on the job. Restore to scratch on a schedule, then check row counts, the newest record's timestamp and a checksum of a known table. Alert on the age of the last successful restore, which is the only number that means anything.
• Alert on silence. A job that stops reporting must page. Absence of failure is not success, and this incident is the canonical demonstration.
• Verify the whole loop including notification. The email path was part of the failure.
• Publish the measured restore time, because an RTO nobody has timed is an aspiration.

Where copying this would be a mistake

Do not schedule a full restore drill of a 50 TB estate weekly because a smaller company needed one. The proportionate design is a continuous cheap check plus a rare expensive one: nightly restore of the smallest complete unit with assertions, a quarterly full restore with a stopwatch, and a documented sampling rule for the rest. The failure being prevented is silent unavailability of recovery, and a small assertive check catches that as well as a large one.

Common weak answers

• "They needed more backups." They had five mechanisms. They had no verification.
• "Stop letting engineers run commands on production." Tempting after a tired-operator incident, and it addresses the trigger rather than the reason the trigger was catastrophic.