advanced 2 min answer

A platform runs three replicas of every service across three availability zones and still experiences total outages. What kinds of failure does that redundancy not address?

zeptoredundancycorrelated-failureblast-radiusdeployment
Show the full answer Hide the answer

What replica redundancy addresses

Independent failures: a host dying, a disk failing, a zone losing power. For those, three replicas across three zones is genuinely effective.

What it does not address

  • Correlated failures from a shared cause. A bad deployment reaches all three replicas. A poisoned configuration reaches all three. A memory leak affects all three at the same rate. Redundancy multiplies copies of the same software, so a software failure is not made less likely by having more of it.
  • Shared dependencies. Three replicas calling one database, one cache cluster, one identity provider have one failure domain regardless of their own count.
  • Overload. Three replicas at capacity fail together, and losing one shifts its load to the others, which accelerates the failure — the classic cascading pattern where redundancy makes the collapse faster rather than preventing it.
  • Data corruption. Replicated data is corrupted in all replicas, quickly and faithfully. Replication is not backup, and this is the confusion that produces the most complete data losses.
  • Control-plane failure, which affects everything simultaneously and is not made better by more instances.

What actually addresses these

  • Staged rollout with automated halt conditions, which is the primary defence against the correlated software failure — the most common cause of total outages in mature systems.
  • Cell-based architecture, partitioning customers into independent cells with their own full stack, so a bad change or a poisonous request affects one cell. This is redundancy applied to the failure domain rather than to the instance count.
  • Capacity headroom that accounts for failure — sizing so that losing a zone leaves enough capacity, rather than sizing to the total.
  • Backups with verified restore, which is the only answer to corruption.
  • Static stability, so control-plane failure does not stop serving.

The question to ask of any redundancy claim

"What is shared?" Three replicas sharing a deployment pipeline, a configuration source, a database and a codebase are three copies of one system. The redundancy is real for the failures that are independent and absent for the ones that are not — and the outages that make the news are almost always the second kind.