Verified Recovery

Backup success rate is near 100% in almost every organisation, including those that discover during an incident that they cannot restore. It measures whether copies were written, which is the easy half.

Verified recovery measures whether a representative sample of backups can actually be restored, correctly, within the stated recovery time objective — continuously, and by automation rather than by an annual exercise.

Why it matters

A corrupt or unreadable backup is indistinguishable from a good one until you try. Silent corruption, encryption key loss, format changes, incomplete captures of dependent systems and undocumented restore prerequisites all produce backups that look perfect and cannot be used.

The RTO clause matters as much as the verification. Restoring a given volume at a given throughput takes a calculable number of hours; if that exceeds the RTO, the design is wrong however good the backups are, and the arithmetic is available before any incident.

Implementation patterns

• Automated restore of sampled backups into an isolated environment, with an integrity check on the restored data rather than merely a successful restore process.
• Measure restore throughput, so the RTO calculation uses observed rather than theoretical numbers.
• Test granular recovery, not only full recovery. Most real requests are one file, one mailbox, one table, one point in time — a design optimised only for full restore serves the rare case and fails the common one.
• Test dependency restoration, since restoring a database without its schema version, its secrets, its network path and its downstream integrations restores something that does not work.
• Immutability with a retention lock the platform's own administrators cannot override, because modern threat models assume the attacker reaches the backups. A backup an administrator can delete is not protection against the scenario that most often needs it.
• Alert on mass-deletion attempts against the backup store, which is a high-signal indicator.

Industry example

Data-protection platforms such as Druva operate at a scale where these properties become architectural rather than procedural. Global deduplication across customers gives large efficiency and introduces two specific hazards: a corrupted shared block affects every backup referencing it, so integrity verification and redundancy on the deduplication store matter disproportionately; and deletion becomes reference counting, where a bug either leaks storage indefinitely or removes data still referenced. Those are where the hardest bugs in such platforms live.

Failure scenarios

• Backups verified only by job status, so corruption is undetected.
• RTO stated without the throughput arithmetic, making it aspirational.
• Full-restore-only design, failing the common granular request.
• Restore that omits dependencies, producing a system that starts and does not work.
• Backups deletable by a compromised administrative credential.
• An annual test, which validates a configuration that has since changed.

Trade-offs

Continuous verification costs compute, storage and engineering to build, and it will occasionally fail for reasons that are not real problems, producing noise.

The alternative is an assumption, and the cost of the assumption being wrong is the entire value of the data. In practice the only serious argument is about sampling rate rather than about whether to verify, and the answer is driven by how much of the estate must be provably recoverable versus statistically likely to be.

Interview question

"Your backup dashboard has been green for two years. What would you do this week to find out whether you can actually recover, and what would you expect to break first?"