Utilisation Target

Queueing delay grows roughly as 1 / (1 − utilisation). The multiplier is about 3.3 at 70% utilisation, 10 at 90%, and 20 at 95%. Latency does not degrade gradually toward saturation; it goes asymptotic.

The consequence is that a utilisation target is not a cost decision. It is derived from the latency requirement, and the resulting idle capacity is the buffer that keeps latency bounded — not waste.

Framing it explicitly matters, because utilisation is the number leadership watches, and "we run at 65% on purpose" needs an argument behind it.

Why variance matters as much as the mean

Queueing delay grows with the variance of service time, not only its average. A workload whose requests differ by orders of magnitude in cost queues far worse at the same utilisation than one with uniform cost.

Two consequences follow, and the second is the more useful:

1. Highly variable workloads need lower utilisation targets.
2. Reducing variance improves latency more than adding capacity does. Separating request classes by expected cost — short and long into different queues or different capacity pools — is frequently the highest-leverage change available, and it costs nothing in hardware.

Implementation patterns

• Derive the target empirically: at what utilisation does the p99 exceed the SLO? That number is the target, and it is usually far below intuition.
• Separate request classes by cost, so expensive work does not queue in front of cheap work.
• Admission control at the knee, refusing work before entering the region where latency explodes. Accepting requests that cannot complete within their deadline produces expensive work nobody is waiting for.
• Continuous batching for accelerator workloads, where finished items leave the batch and new ones join mid-flight, so one long item does not hold the whole batch.
• Report headroom as a deliberate design parameter on capacity dashboards, alongside utilisation, so it is not read as inefficiency.

Industry example

Expensive inference serving illustrates every term at once. Requests differ by orders of magnitude in cost; a long generation occupies an accelerator and blocks shorter requests behind it; and demand is bursty.

Running such a cluster at 90% utilisation produces latency that is unacceptable and unstable, while the utilisation dashboard looks excellent. The workable configuration targets a much lower utilisation, separates interactive from batch traffic into different pools, applies admission control that rejects rather than queues past the deadline, and uses continuous batching to attack the head-of-line blocking term directly.

The same reasoning applies to any capacity plan where latency matters: the honest question is not "how much capacity do we need for N requests per second" but "at what utilisation does our p99 exceed the target".

Failure scenarios

• Capacity planned from throughput, producing a target that meets the request rate and misses the latency SLO.
• Utilisation optimised as a goal, which drives the system into the non-linear region.
• Mixed request costs in one queue, so cheap requests inherit the tail of expensive ones.
• Headroom cut in a cost review, without the latency consequence being modelled.
• Averaging utilisation across a fleet, hiding individual nodes already past the knee.

Trade-offs

Lower utilisation costs money continuously — idle capacity that produces nothing most of the time. For workloads where latency is not user-visible, such as batch processing, running near saturation is correct and the idle capacity would be genuine waste.

The discriminator is whether anyone is waiting. Where they are, the arithmetic that justifies the headroom is queueing theory rather than intuition, and it should be presented that way.

Interview question

"Finance wants your cluster at 90% utilisation to cut costs. It currently runs at 65% and meets its p99 SLO. Explain what will happen and what you would propose instead."