The token stream stays up and the answers get worse
How production LLM serving systems separate prefill from decode, route requests against the KV cache, shed load, and fail.
A field guide to the capacity decision in production LLM serving, reconstructed from the systems of Anthropic, OpenAI, Meta, DeepSeek, Moonshot AI, Character.AI and Perplexity, from three published incident reports, from the vLLM project's own design arguments, and from a Microsoft study of 156 high-severity inference incidents. It answers which of prefill/decode disaggregation, prefix-aware routing and chunked prefill a workload actually needs, what each costs, and which failure will not show up on an SLO dashboard.
The failure that costs most is not the outage: Anthropic's postmortem and Meta's QCon talk independently describe inference bugs that degrade answer quality for weeks while availability, latency and error rates all stay normal.
What you get out of it
- vLLM's own documentation states that disaggregated prefill does not improve throughput; it is a tool for tuning TTFT and tail inter-token latency separately, and Meta measures its cost at 50 to 100 ms of added TTFT.
- Plan on roughly half your input tokens being cache-reusable: DeepSeek measured 56.3% on a day of production traffic and Moonshot AI reports a ceiling near 50%, the only two such figures published from real traffic at scale.
- Two of the three published incidents touching this stack were a telemetry rollout and a quota-policy code path, not GPUs or serving frameworks, and a Microsoft study of 156 incidents puts timeouts and resource exhaustion at the top of the list.
- Admission control that rejects on currently observed load makes prefill and decode saturate in antiphase; Mooncake had to predict downstream load instead, and the naive version is worse than useless.
- Adaptive speculative decoding has been proposed and abandoned repeatedly: vLLM PR 26504 ran eight months and closed as stale, with reviewers arguing acceptance rate is the wrong control signal and goodput is the right one.
Scope
Why this, now. Disaggregated serving moved from research to default architecture across 2025 and 2026, and the marketing case for it now contradicts what its own implementers document.
What it does not cover. Model selection, quantisation and distillation quality, fine-tuning, agent frameworks, retrieval quality, and the buy-versus-build case against a hosted API.