advanced 2 min answer

A platform runs three redundant instances of a component and calculates its availability as extremely high. In practice, all three fail together during incidents. What is the flaw in the reasoning?

redundancycorrelated-failureindependenceconfigurationfastlyfailure-analysis
Show the full answer Hide the answer

The flaw

The arithmetic assumes independence, and the failures are correlated.

Redundancy multiplies availability only when failures are independent. Three instances at 99.9% give 99.9999% if and only if the probability of one failing is unaffected by another failing. In practice they share:

  • The same code. A bug triggered by a particular input affects all three simultaneously. This is the single largest source of correlation and no amount of instance redundancy addresses it.
  • The same configuration. A bad configuration push reaches all three within seconds. Configuration is the most common cause of correlated failure in mature systems, precisely because the mechanism that distributes it is designed to be fast and global.
  • The same dependencies. All three call the same database, the same identity service, the same DNS.
  • The same deployment. A rolling deploy reaches all three; if the new version is bad, redundancy delays the failure rather than preventing it.
  • The same capacity assumptions. If one fails, the other two must absorb its load. If they were sized at 50% utilisation each, losing one takes the survivors to 75% and losing two is unrecoverable.
  • The same physical substrate, if they happen to share a rack, a zone or a power domain.

What redundancy actually protects against

Uncorrelated hardware and infrastructure failure — a disk, a host, a network device. That is a real and worthwhile category, and it is not the category that causes most modern outages.

Most serious incidents come from change: a deploy, a configuration push, a dependency change, a traffic shift. Instance redundancy provides no protection against any of them, which is why systems with excellent redundancy still have outages.

What does protect against correlated failure

Staged rollout with blast-radius limits. Configuration and code reach one instance, then one location, then a region, then everything — gated on health at each stage. This is the primary defence, and it converts "correlated" into "sequential".

Diversity in the things that fail together. Different versions running simultaneously during a rollout, so a bad version is present in a minority. Diversity is expensive and is exactly what redundancy of identical instances does not provide.

Rollback faster than rollout, and independent of whatever broke — including a locally cached last-known-good configuration and an automatic revert on health failure.

Capacity sized for N-1 or N-2, so surviving instances can actually absorb the load.

Cell isolation, so the unit of failure is a defined subset of users rather than a degraded experience for all of them.

The lesson

Count the shared fate, not the instances. The useful question is never "how many replicas do we have?" but "what change or failure would affect all of them at once, and what stands between that change and all of them?" For most systems the honest answer is a deployment pipeline and a configuration system — which is why those deserve more reliability investment than the replicas do.