A company discovers during an incident that five separate backup and replication mechanisms all failed to produce a usable recovery point. What class of failure is this, and what practice prevents it?

The class of failure

Unverified redundancy. Five mechanisms existed. None had been tested end to end recently enough to matter. Each failed for a different, individually mundane reason — a misconfigured flag, a silent version incompatibility, a process that was writing to the wrong destination, a snapshot schedule that had been disabled, an upload that had been failing for weeks with alerts going to an unmonitored address.

The critical insight is that redundancy multiplied the illusion of safety without multiplying the actual safety, because the mechanisms shared a common failure mode: nobody had restored from any of them.

This is not a rare pathology. It is the default state of backup systems, because backups are write-mostly and failures are silent by nature — a backup that does not run produces no error visible to anyone who cares.

Why backup failures are silent

• The failure is an absence, and absences do not generate events.
• Monitoring typically checks that the job ran, not that the output is restorable.
• The people who would notice are the people who never look, because looking means a disaster.
• Success is indistinguishable from failure without an actual restore.

The practices that prevent it

1. Restore testing is the only real test. Automated, scheduled restores into an isolated environment, with verification that the restored system passes real checks — row counts, checksums, application-level smoke tests. A backup that has not been restored is a hypothesis.

2. Measure the restore, not the backup. The metric that matters is time-to-restore and data-loss-at-restore, measured by actually doing it. Backup job success rate is a proxy that fails in exactly the cases you care about.

3. Alert on absence, not only on failure. "No successful backup in N hours" catches the job that stopped being scheduled, which failure alerts structurally cannot.

4. Independence of mechanisms. Different tooling, different storage, different credentials, different region. Redundancy that shares a code path, a credential or a destination is one mechanism wearing five costumes.

5. Immutable, separately-credentialed backup storage. Backups that the production system's credentials can delete are not protection against the most likely modern disaster, which is a compromise or a destructive automation error rather than hardware failure.

6. A published, current RPO and RTO based on measurement. Not aspiration. If the last restore drill took eleven hours, the RTO is eleven hours, whatever the policy says.

The organisational half

The deeper failure is usually cultural: backups are owned by everyone and nobody, checked during audits rather than continuously, and treated as a compliance artefact rather than an operational capability. The fix is to name an owner, put restore time on a dashboard, and run the drill on a schedule that does not depend on anyone remembering.

The uncomfortable question worth asking your own systems

"If the primary database were destroyed right now, who would run the restore, from which artefact, in what time, and when did that person last do it?" If any part of that answer is uncertain, the backups are decorative.