Dark Failover

The first real disaster-recovery test of a system that has never failed over is a negotiation between two true statements: an untested recovery plan is not a plan, and a live failover of a system that has never done one is a scheduled incident with a good chance of becoming an unscheduled one.

Dark failover resolves it by separating the parts that can be measured safely from the part that cannot. The standby is restored, promoted as far as it can be without taking production traffic, scaled to full size, given real credentials and connectivity, and driven with synthetic or internal traffic at production shape. Everything is measured. Nothing user-facing moves.

Why it matters

Most of what a first DR exercise discovers is found before any traffic is cut over. The standard finding is that something essential exists in one region only - an identity provider integration, a certificate, a secret, a third-party allowlist keyed to the primary region's addresses, a licence server. The second standard finding is capacity: standby environments are routinely scaled down to save money, and a recovery that cannot take peak load is not a recovery.

Both are found by a dark failover, at the cost of an environment and a weekend, and neither requires the business to accept risk.

It also produces the number that matters. A measured restore time plus a measured readiness time is a defensible RTO with evidence behind it, which is usually what an auditor is actually asking for, and it is almost always worse than the documented figure - commonly by a factor of 2 to 5 on a first attempt, so a documented 4 hours becomes a measured 8 to 20 hours.

Implementation patterns

• Define "recovered" first, as a named business transaction succeeding end to end from outside the network, at a stated rate, with data no older than the RPO. Without this the exercise ends in an argument about what counts.
• Restore into isolation and time it, before anything touches the standby. Reversible by deleting the environment.
• Drive production-shaped synthetic traffic, not a smoke test. The capacity finding only appears at rate.
• Exercise the human path: the runbook is executed by someone who did not write it, from the document, under a clock.
• Keep the standby read-only on the first pass. A read-only rehearsal is a weaker test and a far safer one, and it is a legitimate first step that avoids any reconciliation.
• Write down which step you are on. Promotion of the standby to accept writes is the point of no return; everything before it is reversible by teardown.
• Publish the measured numbers and the gap to the documented ones, rather than only the pass.

Industry example

The pattern is the safe half of what large organisations do routinely: Netflix's Chaos Monkey and the practice that grew around it established bounded, abortable failure exercises as ordinary engineering, and AWS's static stability guidance makes the companion argument that a standby which must scale up to survive a failure has taken a dependency on the control plane at the worst moment. A dark failover is how you find out whether your standby has that dependency, without discovering it during the event.

Failure scenarios

• A standby that restores but cannot take load, because it was scaled down and the scaling itself needs capacity that may not be available during a regional event.
• Region-scoped credentials, keys or allowlists that nobody documented, surfacing as authorisation errors that look like application bugs.
• Synthetic traffic that is too clean, missing the read and write mix, the cache behaviour and the long-tail endpoints, so the exercise passes and the real event does not.
• A rehearsal that quietly touches production through a shared queue, a shared cache or a shared third-party sandbox, turning the test into the incident.
• Data divergence if the standby is promoted for writes without a reconciliation plan.
• A pass recorded with no measured numbers, which produces confidence and no evidence.

Trade-offs

Dark failover costs a full-size standby for the duration and the engineering time to drive it, and it deliberately stops short of proving the last step: that real traffic, with real clients and their caches and connection pools, converges onto the new location. That last part is exactly where recovery time hides, so a dark failover measures a lower bound and should be reported as one.

The alternative positions are worse. A live failover carries real risk on a system with no history of it, and a tabletop exercise measures nothing at all.

When not to use it

When the system has already failed over in anger or in a drill, the honest exercise is the live one, because the unknowns are now in client convergence rather than in the standby. And for a system whose recovery path is a rebuild from an upstream source - a derived cache, a search index, a reporting copy - rehearse the rebuild instead; failing over a thing you would never fail over is theatre with a budget.

Interview question

Q: An auditor wants evidence for a four-hour RTO on a system that has never failed over, and the business will not accept any risk to production. What do you commit to, what do you measure, and what do you tell the auditor you have not proven?

What a strong answer covers: defining recovery as a user-visible transaction before scheduling anything · restore into isolation, then synthetic production-shaped load, then full readiness with no user traffic · the two standard findings, single-region dependencies and undersized standby capacity · executing the runbook with someone who did not write it · naming promotion-for-writes as the point of no return · and stating plainly that client convergence is unproven, so the measured figure is a lower bound.

Quick check

Quiz: What does a dark failover deliberately not measure? Client convergence - how long real clients take to stop using the old location - which is usually the larger part of user-visible recovery time.

Flashcard: What do first DR exercises usually find? A dependency that exists in one region only, and a standby too small to take peak load.