Utilisation Ceiling

A capacity plan that targets 85% utilisation across three availability zones has already decided to fail. Losing one zone moves its load onto the other two, so each survivor needs to serve 50% more than it was - and at 85% there is nowhere for it to go. The fleet is sized for the day nothing breaks.

The ceiling is arithmetic, not judgement. With N independent failure domains that must survive losing one, the maximum planned utilisation is (N−1)/N: 50% at two domains, 66% at three, 80% at five, 90% at ten. Everything above that line is capacity you are counting on twice.

Why it matters

It converts a reliability requirement into a line on the invoice, which is the only form in which the trade-off can be discussed honestly. At three zones, one third of the fleet exists for an event that may never happen, and someone will eventually ask why. The answer is not "resilience"; it is that the alternative is a total outage the first time a zone is lost, and that the price of avoiding it falls as 1/N.

That 1/N is the strongest argument for many small failure domains rather than a few large ones, and it is the capacity reasoning behind cell-based architectures: twenty cells cost 5% headroom for single-cell loss, where two regions cost 50%.

Implementation patterns

• Express the ceiling as an alert, not a slide. Page when 7-day average utilisation crosses the ceiling, because that is the moment the redundancy became notional.
• Discount for the recovery transient. Survivors absorb reconnection storms, cache warming and re-replication, so a practical target is 55–60% at three zones rather than the theoretical 66%.
• Compute it per constrained resource. A fleet can be at 50% CPU and 95% of its connection limit; the ceiling applies to whatever is scarcest, which is often memory, connections or a licence count rather than CPU.
• Model the zone as unequal. Zones are rarely the same size, so the real requirement is surviving the largest one, not an average one.
• Where provisioning is fast and the workload stateless, plan for degraded-then-recovered rather than full standby, and write down the minutes of degradation you have accepted.

Industry example

Published cloud architecture guidance has made this static-stability argument for years: a system should keep working during a zone failure using capacity it already holds rather than depending on a control plane to give it more, because the control plane is itself under stress during the event that made you need it. The same arithmetic shows up in the planning rules for quorum systems, where a five-node group tolerates two losses and a three-node group one, and the headroom follows the same (N−1)/N shape.

Failure scenarios

• Correlated failure defeats the arithmetic. If all three zones run the same unattended OS update, as Datadog has described from its March 2023 multi-region incident, they are not three domains and the ceiling was never real.
• The ceiling is held on CPU while connections are the constraint, so failover fails on a resource nobody was watching.
• Autoscaling masks it. A fleet that scales reactively looks comfortable until a zone loss demands capacity faster than instances can boot and warm.
• Quota, not capacity, binds. The survivors have room in the plan and no headroom in their account limits, which is a five-minute failure and a two-hour ticket.

Trade-offs

Running under the ceiling costs real money every day: at three zones, 33% of a fleet is idle insurance. Running above it converts a single-domain failure into a full outage. The middle path is explicit degradation: plan above the ceiling for non-essential paths and under it for the critical ones, so a zone loss sheds the recommendation feed rather than checkout. That requires knowing which paths are which, written down before the event.

When not to use it

For a batch workload with no latency obligation, the right answer is to run at 95% and let a zone failure delay the job - the ceiling is a liveness requirement, not a law of nature. For an internal tool with a handful of users, paying 33% headroom to avoid an hour of unavailability is poor value, and a documented recovery procedure is enough. The ceiling also stops being the right model when the instance is the failure domain rather than the zone, because at hundreds of instances the headroom term is negligible and the binding question becomes how fast replacements become useful.

Interview question

Q: Finance asks why a three-zone fleet runs at 55% average CPU. Make the case, then tell me what you would change to run it at 75% without accepting a worse outage.

What a strong answer covers: the (N−1)/N arithmetic and why 55% is the practical form of 66%; the recovery transient that consumes the difference; options that genuinely raise the ceiling (more and smaller failure domains, a degradation ladder that sheds non-critical load on failover, pre-warmed replacement capacity, quota checked in advance); which resource is actually scarce; and the honest admission that correlated failures - shared updates, shared control planes, shared dependencies - make the number optimistic.

Quick check

Quiz: What is the maximum planned utilisation for a fleet across five zones that must survive losing one? About 80%, from (5−1)/5, less a margin for the recovery transient.

Flashcard: Why do many small cells cost less redundancy than two regions? Headroom for single-domain loss falls as 1/N, so twenty cells need 5% and two regions need 50%.