An inference platform receives computationally expensive requests in unpredictable bursts. Should it use admission control, request queues, dynamic batching, autoscaling, priority classes or pre-provisioned capacity?

What each contributes, and their order

• Dynamic batching first, because it is the largest throughput lever available. Combining concurrent requests into one forward pass is frequently a several-fold improvement on the same hardware, at a modest latency cost. Nothing else in this list changes capacity by that factor.
• Admission control second. A hard limit on in-flight work, sized so that accepted requests can complete within their deadline. Excess is rejected quickly and explicitly, because a slow rejection causes a client timeout and a retry, which multiplies load exactly when you are shedding it.
• Queues for latency-tolerant work, which smooths demand into the troughs. Free utilisation, and it must be bounded — an unbounded queue converts a capacity problem into a memory problem and then into an outage.
• Priority classes, so an interactive request is not stuck behind a batch job. This is also the natural product boundary.
• Pre-provisioned warm capacity for the popular subset, since usage is heavily skewed and a small warm pool covers most traffic.
• Autoscaling last, as a slow backstop. With cold starts measured in tens of seconds it cannot respond to a burst, and treating it as the primary control is the common mistake.

The lever that improves all of them

Reducing the cold start itself. Snapshot and restore of process memory, lazy image pulls, node-local weight caches, and affinity routing — sending a request for model X to a node already holding X — which converts most cold starts into warm ones without holding anything idle.

This is engineering paid once against a warm-capacity cost paid continuously, and for expensive accelerators the arithmetic favours it almost immediately.

The trade-off to expose rather than resolve

Batching improves throughput and worsens latency, and the correct batch window differs by workload. Rather than choosing one, mature platforms expose it as priced tiers: a guaranteed-latency class backed by warm capacity with small batches, and a cheaper best-effort class with larger batches and queueing.

That moves the decision to the customer, who knows whether their workload is interactive — and converts an internal dilemma into revenue segmentation.