Cloud Disaster Recovery
Designing and rehearsing recovery from the loss of a zone, a region or a provider — where the plan is worth exactly as much as its last test.
Definition
Disaster recovery is defined by two numbers agreed with the business, not by engineering preference:
- RTO — how long until service is restored.
- RPO — how much data may be lost, measured in time.
Everything else follows from those two, and both must be per-service. An RTO of four hours for reporting and fifteen minutes for checkout is a coherent position; a single number for the whole estate is not.
The strategies and their honest costs
| Strategy | RTO | RPO | Cost |
|---|---|---|---|
| Backup and restore | Hours to days | Hours | Lowest |
| Pilot light (core services minimal, scale on failover) | Tens of minutes | Minutes | Low |
| Warm standby (scaled-down full copy) | Minutes | Seconds | Moderate |
| Active-active | Near zero | Near zero | Highest, plus data-consistency complexity |
Active-active is frequently assumed to be the goal. It is the right answer only when the business value of near-zero RTO exceeds the substantial permanent cost — not just infrastructure, but the conflict resolution, routing, and testing burden of accepting writes in two places.
What the strategy table does not tell you
The dependency you cannot fail over. Identity, DNS, secrets management, the CI/CD pipeline, the container registry and the observability stack. If those are single-region, a regional failover cannot be executed — you will be unable to deploy, authenticate or see anything during the exact window in which you need all three. This is the most common gap in otherwise thorough DR plans.
Failover is not the hard part; failback is. After a region returns, reconciling the data written during the outage is where the genuine complexity lives, and it is almost never rehearsed.
Capacity in the second region must actually exist. During a large regional event, everyone else is also failing over, and on-demand capacity may not be available. Reserved capacity is the only guarantee.
Industry example
Widespread regional cloud outages have repeatedly demonstrated the same pattern: organisations discover their dependency graph during the incident rather than before it. A service is multi-region, but its configuration store is not; a failover runbook exists, but the identity provider it authenticates against is in the failed region; a standby is ready, but nobody has permission to promote it because the approval tool is also down.
The transferable practice is rehearsal that includes the control plane — not a test of whether the standby can serve traffic, but of whether your organisation can execute the switch under the conditions in which it would be needed.
Failure scenarios
- Backups never restore-tested, so recovery time is unknown until it is measured in production.
- The runbook stored in the system that is down.
- Asynchronous replication with an undocumented RPO, so the data loss on failover is a surprise.
- A standby that has drifted because deployments only ever went to primary.
Interview question
"Walk me through a regional failover. Name every dependency that must already exist in the second region for the plan to be executable."