A marketplace runs across three availability zones in one region. What actually happens when one zone fails, and which common design choices turn a survivable zone failure into an outage?

What a zone failure means

Not a clean power-off. Zone failures are usually partial and messy: elevated packet loss, degraded storage latency, some instances unreachable while others respond slowly, and control-plane operations in that zone failing intermittently. A design that assumes a clean binary failure is designing for the easy case.

The design choices that turn it into an outage

1. Capacity sized so all three zones are needed. If steady-state load requires 100% of three zones, losing one leaves 67% of required capacity, and the survivors fail under the shifted load. Surviving a zone failure means running at roughly two-thirds utilisation — which is the single most commonly cut cost, and the reason zone redundancy so often does not work when needed.

2. Stateful components with a quorum that cannot survive the loss. A three-node cluster with one node per zone survives one zone. A cluster with two nodes in one zone and one in another does not survive losing the first zone, even though it is "spread across zones". Quorum placement matters more than distribution.

3. Cross-zone chattiness. Services making many cross-zone calls per request see latency multiply when a zone degrades. Zone-local routing with cross-zone fallback keeps the common path fast and contains the damage — and reduces cross-zone data transfer costs as a side benefit.

4. Health checks that cannot distinguish slow from dead. A degraded zone that responds slowly may pass health checks and keep receiving traffic, which is worse than being marked dead. Health checks need latency thresholds, not just liveness.

5. Deployment and control-plane dependencies in one zone. If the deployment system, the configuration store, or the secrets service lives in the failed zone, you cannot respond. Everything needed to react must survive.

6. Single-zone dependencies nobody documented. A cache, a queue, a licence server, a legacy component someone put in one zone years ago. These are found during the incident, and they are the most common cause of a "multi-AZ" system failing on a single-zone event.

What good looks like

• Capacity headroom sized for N-1, stated explicitly as a cost decision rather than discovered.
• Static stability: the surviving zones handle the load without needing to scale, because scaling requires a control plane that may itself be degraded.
• Zone-aware routing that automatically drains a zone on health signals.
• Regular zone-evacuation drills. A zone failure you have practised is an event; one you have not is an outage.
• An inventory of single-zone components, maintained as a fitness check rather than as tribal knowledge.

The lesson

Multi-zone deployment is not a property you get by placing instances in three zones. It is a property you get by sizing for N-1, placing quorums correctly, removing single-zone dependencies, and practising the evacuation. Most systems described as multi-AZ have the placement and none of the rest.