metric

Reliability Headroom

The gap between provisioned capacity and the load that would be carried after the largest planned failure, measured under peak conditions.

Ordinary capacity planning asks whether there is enough capacity for peak demand. Reliability capacity planning asks whether there is enough after losing a fault domain, at peak, with the load redistributed.

The arithmetic that catches teams out: three zones at 70% utilisation. Lose one, and the surviving two must carry 105% — so they saturate, latency climbs, timeouts trigger retries, and the retries finish them. Redundancy that cannot absorb the redistributed load is a slower cascade, not protection.

The ceiling that follows: with N zones, steady-state utilisation must stay below (N−1)/N. Three zones means below 67%; two zones means below 50%, which is why two-zone deployments are expensive per unit of resilience.

Autoscaling is not a substitute. Redistribution is instantaneous; scale-up takes minutes, and in a zone failure the control plane may itself be degraded. Pre-provisioned headroom is the statically stable choice.

Two refinements: size for peak plus failure, not average plus failure, since failures do not avoid busy periods; and verify the headroom by load testing a degraded configuration rather than calculating it — connection pool limits and downstream quotas frequently bind before CPU does.