An inference service's utilisation rises from 70% to 90% and latency more than triples. Why is the relationship non-linear, and what does that imply for capacity planning?

Why the relationship is non-linear

Queueing delay grows roughly as 1 / (1 − utilisation). At 70% utilisation the factor is about 3.3; at 90% it is 10; at 95% it is 20. Latency does not degrade gradually as you approach saturation — it goes asymptotic.

The intuition is that arrivals are not evenly spaced. Even with average arrival rate below capacity, random clustering means a queue forms, and the higher the utilisation, the less idle time exists to drain it before the next cluster arrives.

Why it is worse for this workload

Two factors amplify it severely for expensive inference:

1. High service-time variability. Requests differ by orders of magnitude in cost — a short completion versus a long generation. Queueing delay grows with the variance of service time, not only its mean. A workload with highly variable request cost queues far worse than one with uniform cost at the same utilisation.

2. Head-of-line blocking. A long-running request occupying an accelerator blocks shorter requests behind it. Without separation, a few expensive requests dominate the latency of many cheap ones.

What it implies for capacity planning

1. Target utilisation well below saturation. For latency-sensitive workloads with variable service times, that frequently means 60–70% rather than 90%. The idle capacity is not waste; it is the buffer that keeps latency bounded, and framing it that way is necessary because utilisation is the number leadership watches.

2. Reduce variance, not just mean. Separating request classes by expected cost — short and long into different queues or different capacity pools — improves latency more than adding capacity does. This is usually the highest-leverage change available.

3. Plan capacity on the latency target, not on throughput. "How much capacity do we need to serve N requests per second?" is the wrong question. The right one is "at what utilisation does our p99 exceed the target?", which is a much lower number.

4. Admission control at the knee. Since latency explodes past a threshold, the system must refuse work before entering that region. Accepting requests it cannot complete within their deadline produces expensive work nobody is waiting for.

5. Continuous batching as the mechanism that changes the economics — finished sequences leave the batch and new ones join mid-flight, so a long request does not hold the whole batch. This directly attacks the head-of-line blocking term.

The lesson

Utilisation targets are set by latency requirements, not by cost efficiency. A cluster running at 95% with unacceptable p99 is delivering less value than one at 70% meeting its target — and the arithmetic that justifies the idle capacity is queueing theory, not intuition.