Shared-Cause Analysis

Three replicas across three zones is genuine protection against a host dying, a disk failing or a zone losing power. It is no protection at all against anything that reaches all three.

The audit question is simply: what is shared?

Why it matters

The outages that take a whole system down are almost never independent hardware failures — those are handled well by every framework. They are bad deployments, poisoned configuration, overload, data corruption and control-plane failure, all of which reach every replica simultaneously.

Teams reason about redundancy in terms of instance count and conclude they are protected against a class of failure that instance count does not address.

Implementation patterns

Enumerate what all replicas share, then address each:

• The same code, via the same pipeline. Addressed by staged rollout with automated halt conditions, which is the primary defence against the most common cause of total outages in mature systems.
• The same configuration source. Addressed by validation before propagation, staged config rollout, and a rollback path that does not depend on the pipeline being rolled back.
• The same downstream dependencies. Three replicas against one database is one failure domain; addressed by bulkheads, fallbacks and, at the limit, cell-based partitioning.
• The same load. Replicas at capacity fail together, and losing one shifts load to the others, accelerating the collapse. Redundancy makes the cascade faster rather than preventing it unless headroom is sized for post-failure capacity.
• The same data. Replication propagates corruption faithfully and quickly. Replication is not backup, and this confusion produces the most complete data losses.
• The same control plane. Addressed by static stability.

Industry example

Fast-growing platforms such as Zepto that add zones and replicas as they scale routinely find that their availability did not improve proportionally, because the failures they actually experienced were deployments and configuration rather than infrastructure. The intervention that changes the outcome is cell-based partitioning plus staged rollout — redundancy applied to the failure domain rather than to the instance count.

Failure scenarios

• Instance count treated as the availability measure.
• A bad deployment reaching every replica within minutes, with no automated halt.
• Headroom sized to total rather than post-failure capacity, so a single instance loss cascades.
• Replication mistaken for backup, leaving no recovery from corruption.
• A shared control plane on the request path, making its outage total.

Trade-offs

Addressing shared causes is more expensive than adding replicas, and considerably less visible. Cell-based architecture in particular multiplies operational surface: more deployments, more monitoring, more places for configuration to drift, and cross-cell operations that were previously trivial.

The judgement is that replicas are cheap and address the failures you rarely have, while cells are expensive and address the failures that actually take you down. For a system whose outage history is dominated by deployments and configuration, the arithmetic clearly favours the cells — and the outage history is the evidence to bring.

Interview question

"You run three replicas in three zones and had two total outages last year. Neither was infrastructure. Tell me what your redundancy was actually protecting against, and what you would build instead."