Correlated Region Failure

A business case for a second region is usually built on multiplication: if one region is available 99.9% of the time, two independent regions give 99.9999%. The arithmetic is correct and the premise is false. The regions are not independent, and the dependencies they share are the ones most likely to cause an outage in the first place.

Correlated region failure is the set of those shared dependencies: the same application build, the same configuration push, the same schema migration, the same identity provider, the same DNS zone, the same deployment pipeline and frequently the same globally-scoped control plane. A second region protects against the failure of a region. It does nothing about the failure of a change, and changes cause the majority of incidents for most services.

Why it matters

The cost is concrete: a second region roughly doubles infrastructure and more than doubles operational load, and the availability it buys is often one additional nine rather than three. Teams that commit to it on the independence assumption discover afterwards that their top incident causes — a bad deploy, a bad config, a bad migration, a dependency outage — are all region-agnostic.

The second-order harm is worse. An untested failover path is a liability, because the organisation believes it has redundancy and plans around it. The mechanism is the same one that makes warm standby fail: recovery depends on control-plane actions that are themselves degraded, so the failover either does not complete or lands traffic on capacity that cannot serve it.

Implementation patterns

• Enumerate the shared fate list explicitly: build artefacts, config plane, schema versions, identity, DNS, TLS issuance, secrets, the pipeline, and any service with a single global endpoint. Anything on that list is a failure the second region does not cover.
• Decorrelate the change path first, because that is where the correlation is cheapest to break. Deploy to one region, bake for a defined period, then the other. Same for configuration and schema.
• Make the regions statically stable: pre-provisioned at the size failover needs, so recovery requires no scaling or launching. This is why serious multi-region is expensive.
• Run the failover on a schedule, in production, not as a tabletop. A quarterly exercise is the only evidence that the path works.
• Measure correlation from your own incident history: of the last twenty incidents, how many would a second region have prevented? The number is usually between 10% and 30%, and it is the honest input to the business case.

Industry example

The best-documented cases involve a change propagating globally rather than a region dying. A global edge network's July 2019 outage came from a single firewall rule with catastrophic regex backtracking, pushed everywhere at once, which pegged CPU across the fleet for about 27 minutes — no amount of geographic redundancy helps, because the bad thing was deployed to all of it. A second edge provider's June 2021 outage followed the same shape: a valid customer configuration triggered a latent bug from an earlier deployment and most of the network returned errors for under an hour. Both are region-independent failures in estates with extreme geographic redundancy, and both were resolved by rolling back a change rather than by shifting traffic.

Failure scenarios

• Simultaneous bad deploy to both regions, because the pipeline treats them as one environment.
• A schema migration applied to a replicated dataset, which reaches every replica by design.
• Global control-plane degradation, where failover automation cannot obtain capacity in the standby.
• A shared identity or DNS dependency failing, which takes down both regions while both look healthy internally.
• Failover to a region whose capacity was never tested, converting a partial outage into a total one.
• Data divergence after an unplanned failover, when both sides accepted writes and reconciliation takes far longer than the outage did.

Trade-offs

The third row is the common one and the worst value in the table.

When not to use it

Multi-region is justified by data residency, user latency or a contractual recovery objective a restore cannot meet — rarely by an availability target on its own. If the target is 99.95% and the incident history shows deploys and dependencies rather than regional events, the cheaper nine comes from staged rollouts, better tests and faster rollback. A second region taken on before the change path is decorrelated buys mostly cost. The flip is clear: when a regulator requires data to stay in a jurisdiction, or when a single-region restore would take 12 hours against a 1-hour objective, the argument is settled without any availability arithmetic.

Interview question

Q: A stakeholder wants multi-region for availability. Your service is at 99.93% and the target is 99.99%. What do you look at before agreeing, and what would make you say no?

What a strong answer covers: classify the last year's downtime by cause and count how much a second region would have prevented; expect most of it to be change-related. Compare the cost of decorrelating the change path — staged rollouts, faster rollback, canaries — against roughly 2x infrastructure plus a second operational surface. Say no when the residual regional risk is a small share of downtime and there is no residency or recovery-objective requirement. Say yes without argument when residency or a contractual RTO decides it, and then insist on static stability and scheduled failover exercises, because an untested second region makes the number worse rather than better.

Quick check

Quiz: Why do two 99.9% regions not give 99.9999% availability? — Because the failures are correlated: the same build, config push, schema change, identity and DNS reach both regions, and change-related failures dominate most services' downtime.

Flashcard: What is the first thing to decorrelate when moving to two regions? — The change path. Staged deploys, configuration and schema migrations region by region, with a bake period, because a simultaneous bad change defeats geographic redundancy entirely.