advanced 2 min answer

Your platform runs across three availability zones. How would you determine whether it actually survives losing one?

capacityzone-failureheadroomtesting
Show the full answer Hide the answer

Start with the arithmetic, because it usually fails there

Steady-state utilisation must stay below (N−1)/N. Three zones means below 67%.

At 70%, losing one leaves two carrying 105%: they saturate, latency climbs, clients time out and retry, and the retries finish them. Redundancy that cannot absorb the redistributed load is a slower cascade, not protection.

Check it at peak plus failure, not average plus failure. Failures do not avoid busy periods, and sizing against the average is the most common version of this mistake.

Autoscaling does not rescue it

Redistribution is instantaneous; scale-up takes minutes. Worse, in a zone failure the control plane may itself be degraded — and scaling is a control-plane operation.

Pre-provisioned headroom is the statically stable choice, and its cost is honest: lower utilisation and a higher bill. That is the price of not depending on a dependency during its most likely failure.

Find the non-obvious constraints

CPU is rarely what binds first. Check connection pool sizes (a pool sized for one third of traffic does not absorb one half), downstream quotas and rate limits, per-instance concurrency caps, NAT gateway port allocation, and cross-zone data transfer cost, which is a real budget surprise in a sustained degraded state.

Check the data tier separately

Stateless services redistribute. Databases do not. Ask specifically: is the primary in the failed zone, and if so is promotion automatic? Is quorum still satisfiable with two zones? Is replication synchronous across zones, and what does losing one do to write latency?

Quorum systems with three replicas across three zones survive one zone loss and cannot tolerate a second — that is worth stating explicitly, since it is often assumed to be more resilient than it is.

Then test it rather than trusting the analysis

Load test a degraded configuration — take one zone out and drive peak traffic through the remainder. Everything above is a hypothesis until this runs.

Progression: non-production first, then production during a low-traffic window with an abort, then production at higher traffic once it has passed.

What a strong answer adds

Naming the assumption behind the whole design: that zones fail independently. Shared control planes, a shared configuration store, a single-zone dependency, or a global service used by all three break that independence — and none of them appear in the capacity model.