A platform runs its worker fleet at 85% average utilisation to control cost. Queue times are becoming unpredictable. What does queueing theory say is happening?

The relationship

Wait time is proportional to utilisation / (1 − utilisation). That factor is roughly 1 at 50% utilisation, 4 at 80%, 5.7 at 85%, 9 at 90% and 19 at 95%. Between 80% and 95% utilisation, wait time increases by a factor of five while the utilisation number moves by an amount that looks small on a dashboard.

This is why 85% feels like a reasonable efficiency target and produces unpredictable latency: it is already on the steep part of the curve, and any variance in arrival rate pushes it further.

What makes it worse

Variability, in both arrivals and service times. The formula above assumes a particular arrival pattern; real arrivals are bursty and real service times have long tails, and both multiply the wait. A workload with highly variable job durations — a two-minute test suite and a forty-minute one in the same queue — behaves far worse at a given utilisation than one with uniform durations.

This is the practical reason to separate workload classes: it reduces variance within each queue, which improves wait times at the same utilisation.

What to do

• Target a utilisation consistent with the latency requirement, not with a cost target. For latency-sensitive work that is often 60–70%; for batch work where waiting is acceptable it can be far higher.
• Separate classes to reduce variance within each queue, and use short-job-first where fairness permits, since it minimises average wait.
• Bound the queue and shed beyond it, because an unbounded queue converts a latency problem into a memory problem and then into an outage.
• Publish queue position and estimated start, so the caller can decide rather than waiting blindly.
• Reduce service-time variance where possible — caching, warm pools, timeouts on runaway jobs — which improves latency at unchanged utilisation and is the cheapest available win.

The framing that resolves the cost argument

High utilisation and low latency are alternatives, not goals to be jointly optimised. The choice is a product decision, and the honest version of the conversation is "we can run at 85% and accept unpredictable waits, or at 65% and be predictable" — with the cost difference stated. Teams that do not frame it this way end up defending an SLA they have made mathematically unachievable.