Admission Control and Load Shedding

A web server under 2x load makes every request 2x slower, roughly, and recovers when the spike passes. An LLM server under 2x load behaves worse than that, because requests are not interchangeable units of work: each admitted sequence claims KV-cache memory for its lifetime, prefill and decode fight for the same GPU, and a request's total cost is unknown at arrival since nobody knows how many tokens it will generate. Admit too much and the scheduler starts preempting running sequences to reclaim cache blocks, recomputing or swapping their state; throughput as measured in tokens per second can stay high while every individual stream's TPOT sails past its SLO. The fleet is busy and the product is broken, which is the signature failure goodput accounting exists to name (see serving SLOs).

Saying no at the door

Admission control moves the rejection to the cheapest possible point: before the request touches a model server. The primitive signals are queue depth and wait time (a request that will queue for 20 seconds against a 2-second TTFT SLO is already failed; reject it now with a 429 and Retry-After so the client backs off instead of retrying into the spike), and token-aware capacity accounting rather than request counting. Prompt tokens are known at arrival and give the prefill cost; decode cost is unknowable, so systems bound it with max_tokens, predict it from request features, or admit optimistically and correct with preemption. Engines expose the same idea internally: vLLM's max_num_seqs and max_num_batched_tokens cap concurrent sequences and per-step batched tokens, which is admission control at the scheduler level; chunked prefill exists for the related problem of one admitted long prompt stalling every running decode (Agrawal et al., OSDI 2024, arXiv:2403.02310).

One subtlety separates LLM admission from classic web admission: rejecting a request mid-conversation is far more expensive to the user than rejecting it at conversation start, and a rejected retry that lands on a different replica loses its prefix cache, making the retry more expensive than the original. Admission policy, session affinity, and prefix-aware routing have to be designed together.

Priorities, and degrading instead of refusing

Not all traffic deserves the same door. The standard split is interactive versus batch: batch and background work (evals, embeddings backfills, agent housekeeping) absorbs arbitrary delay, so under pressure it is shed first, or preempted where the engine supports priority scheduling. Interactive traffic then degrades along a menu that trades quality for capacity rather than availability: route to a smaller or distilled model, shrink retrieved context, cap output length, disable extended thinking, drop optional enrichment passes. Each step frees compute while keeping an answer flowing, and each is a product decision wearing an infrastructure costume, which is why the degradation ladder should be written down and agreed before the incident rather than improvised at 3 a.m. inside a gateway config.

Autoscaling is the partner discipline, not the substitute: new GPU capacity arrives on cold-start timescales of minutes, so admission control owns the first minutes of every spike regardless of how good the scaling policy is (see autoscaling and cold starts).

When it breaks

• GPU utilisation lies to the shedder. A server at 90% utilisation may be healthily saturated or thirty seconds from KV exhaustion; shed on queue delay, batched-token backlog, and cache occupancy, never on utilisation.
• Uniform shedding punishes the wrong users. Random 429s hit a paying tenant's checkout flow and a hobbyist's batch job with equal probability; sheds without priority classes convert an infrastructure incident into a revenue incident.
• Preemption can cascade. Under memory pressure, preempting a sequence to admit another triggers recomputation that consumes the very capacity the preemption was meant to free; bounded preemption budgets and admission caps must be tuned jointly.
• Clients defeat polite backpressure. SDKs with aggressive retry loops turn one shed into three arrivals; without jittered exponential backoff honoured client-side, admission control measures its own echo.
• The degradation ladder drifts. The small fallback model ages while the primary is upgraded, and the quality gap widens silently until an incident routes real traffic onto it; evaluate the degraded paths on the same cadence as the primary.