An inference platform receives a burst of expensive long-context requests that would starve short interactive ones. Design admission control, priority classes, token-based limits and batching so interactive latency SLOs hold.

Why request-count limits do not work

Requests are not fungible in this workload. One request with a 100,000-token context and a 4,000-token output can consume more compute and more GPU memory than a thousand short interactive turns. A rate limiter counting requests permits a tenant to consume the entire fleet while remaining well inside its quota.

Everything must be denominated in tokens — or better, in an estimated cost combining prefill tokens, decode tokens and cache residency, since those consume different resources.

Priority classes

• Interactive: a human is waiting. Strict latency SLO, especially on time-to-first-token, which is what the user perceives.
• Batch and background: summarisation jobs, evaluations, offline enrichment. No latency requirement, high throughput requirement, and deferrable by hours.
• Bulk/long-context: large document processing. Expensive, and usually not latency-critical.

Separate queues per class, with reserved capacity for interactive, so that no volume of batch work can consume the interactive allocation. A single queue with priority ordering is insufficient, because a long-running request already admitted continues to occupy memory regardless of what arrives behind it.

Admission control that reflects the actual constraint

Admission must be governed by KV-cache memory, not by request count, because memory is what limits concurrency. Before admitting a request, estimate its cache footprint from its context length and its likely output length, and admit only if the memory is available.

• Reject or queue when the projected memory would exceed the budget, rather than admitting and thrashing.
• Reserve a memory floor for interactive traffic, always.
• Cap maximum context length per priority class, so a single request cannot monopolise an instance.
• Estimate output length from the request or from historical behaviour, since a request that generates 4,000 tokens occupies its memory far longer than one generating 40.

Batching and preemption

• Continuous batching so completed sequences leave and new ones join at each step, raising utilisation without adding latency for short requests waiting behind long ones.
• Chunked prefill, so a very long prompt's prefill is broken into pieces interleaved with decode steps for other sequences — which prevents one enormous prompt from stalling every in-flight interactive response, and is one of the highest-value mechanisms here.
• Preemption with cache offload for the lowest-priority requests: their KV cache is moved to CPU memory and restored later. Rescheduling is far cheaper than restarting, and it makes priority meaningful after admission rather than only at it.
• Separate pools for prefill and decode, so a burst of long prompts saturates the prefill pool without affecting the decode latency of established sessions.

Per-tenant fairness

• Token-denominated quotas, with a burst allowance.
• Weighted fair queueing over available capacity rather than hard caps, so a tenant can exceed its share when the fleet is idle and is constrained only under contention. Higher utilisation and more headroom simultaneously.
• Separate quotas for interactive and batch, with a documented way for a tenant to declare which they are sending — a batch job that identifies itself can be given a large quota at low priority, which is better for everyone than pretending it is interactive.

What the client must be told

Time-to-first-token and inter-token latency as separate SLOs, since they are governed by different mechanisms. A queue position or an estimated wait when the request is queued rather than rejected. And explicit, retryable overload responses, so that clients back off rather than retrying immediately and amplifying a capacity problem into an outage.