advanced 2 min answer

You plan a regional failover. Name every dependency that must already exist in the second region for the plan to be executable, not just for traffic to be served.

drfailovercontrol-planedependenciesrehearsal
Show the full answer Hide the answer

What is being tested

Whether you think about the control plane as well as the data plane. Most DR plans verify that the standby can serve traffic and never verify that the organisation can execute the switch.

The data plane (usually planned)

  • Compute capacity, actually reserved rather than assumed available — during a large regional event everyone else is failing over too, and on-demand capacity may not exist.
  • Data replicated with a known, monitored RPO.
  • Load balancers, and DNS or global routing able to shift traffic.
  • Caches, which will be cold — so the origin must survive a cold-cache thundering herd, or the failover causes a second outage.

The control plane (usually missing)

This is the list that separates a rehearsed plan from a documented one:

  • Identity provider. If authentication is single-region, nobody can log into anything — including the tools needed to perform the failover.
  • Secrets management. Applications starting in the second region need credentials. If the vault is in the failed region, nothing starts.
  • DNS control. Not just DNS resolution — the ability to change records.
  • CI/CD and the artefact registry. During an incident you will need to deploy a fix. If the pipeline or the container registry is single-region, you cannot.
  • Observability. If dashboards, logs and alerting are in the failed region, you are operating blind at precisely the moment you need to see.
  • The runbook itself, and the communication channels used to coordinate. A runbook stored in a wiki hosted in the failed region is a recurring and entirely predictable failure.
  • Approval and access tooling. If promoting a standby requires an approval in a system that is down, the plan is blocked on process rather than technology.
  • Third-party dependencies, which may be region-pinned in ways you have not checked.

Failback, which is harder than failover

Reconciling data written during the outage — and deciding what to do about writes that occurred in both regions if the failure was partial — is where the genuine complexity lives, and it is almost never rehearsed. A plan that covers failover and not failback covers half the incident.

What makes a plan real

A rehearsal that includes the control plane, run regularly, with the primary genuinely unavailable rather than simulated. Anything less tests whether the standby works, not whether you can use it.

And an RTO and RPO per service, agreed with the business. Four hours for reporting and fifteen minutes for checkout is a coherent position; one number for the whole estate is not, and it leads to either over-spending everywhere or discovering during an incident that the numbers were aspirational.