The token stream stays up and the answers get worse: LLM inference capacity and routing

The territory

The problem, stated without naming the technology, and the systems that have solved it in public.

You run a service where every request has two phases that want opposite things from hardware. The first phase reads the whole input at once and is limited by arithmetic. The second phase emits the answer one step at a time, is limited by memory bandwidth, and holds a per-request working set that grows with every step and that you cannot evict without redoing the first phase. Requests arrive in bursts. The working set is the scarcest resource you own. Two requests that share the beginning of their input could share most of that working set, but only if they land on the same machine, and your load balancer does not know that.

Everything in this guide follows from those five sentences. Prefill and decode want different parallelism, so teams split them onto different pools. The KV cache is the capacity constraint, so teams tier it across HBM, DRAM and flash and route requests toward whichever replica already holds the prefix. Both moves add network hops and coordination, and both change what failure looks like. The interesting part is that none of this is contested at the level of mechanism. It is contested at the level of whether it is worth it for you.

56.3%

of DeepSeek's input tokens over 24 hours were served from the on-disk KV cache rather than recomputed

DeepSeek, Feb 2025

16%

of Sonnet 4 requests misrouted at the worst hour of Anthropic's August 2025 degradation, with no outage

Anthropic postmortem, Sep 2025

~60%

of high-severity incidents at a hyperscale inference service were inference-engine failures, dominated by timeouts

Microsoft, Oct 2025

50–100ms

added to time-to-first-token by splitting prefill from decode, measured at Meta

QCon SF, May 2025

The finding that changed how I read the rest

The failure that costs the most is not the outage. Anthropic's September 2025 postmortem describes three bugs that degraded output quality for roughly a month while availability stayed normal, and states plainly that "the evaluations we ran simply didn't capture the degradation users were reporting, in part because Claude often recovers well from isolated mistakes". Meta's inference lead, describing an unrelated system at QCon four months earlier, put the same thing structurally: "inference bugs can manifest as subtle performance degradation, because LLMs are probabilistic models… you have something horribly wrong, but the result comes out still decently correct". Two independent accounts, two different stacks, the same conclusion: the serving layer has a failure mode that your SLO dashboard is structurally incapable of seeing.

What this guide covers. Self-hosted or self-operated serving of open-weight and proprietary models where you own the capacity decision: how prefill and decode are separated, how the KV cache is tiered and routed against, what the published cost and hit-rate numbers actually are, and what has broken. What it deliberately does not cover: model selection, quantisation and distillation quality trade-offs, fine-tuning, agent frameworks, retrieval quality, or the buy-versus-build case against a hosted API. Those are separate decisions and pretending they are one decision is how architecture documents get long and useless.

Figure 1 · Who has published what, and at which layer

The design layer is crowded and the failure layer is nearly empty: three published incident reports touch this stack, and none of them is about disaggregated serving itself. Sources: Anthropic, OpenAI, Google Cloud, Mooncake, vLLM RFC 10818.

Diagram source

How it is actually built

The common shape across five published systems, and the three places where they genuinely diverge.

Every system in the evidence set converges on the same five-part shape, and the convergence is worth noticing because these teams did not copy each other. DeepSeek published its layout in February 2025 as part of an open-infrastructure week; Moonshot AI published Mooncake as a paper in June 2024; Meta described its stack at QCon in May 2025; the vLLM and llm-d communities argued theirs into existence in public issue threads. They arrive at: an admission gate, a cache-aware router, a prefill pool, a decode pool, and a KV cache tiered across memory it does not own exclusively.

Figure 2 · The reference architecture, with divergence points marked

The KV cache is the centre of gravity, not the model weights: the router exists to steer requests toward cache that already exists, and the pool split exists to stop one phase from stalling the other. Sources: Mooncake, 2024, DeepSeek, 2025, Red Hat on llm-d, 2025.

Diagram source

The admission gate is load-bearing

Mooncake's authors are explicit that they cannot assume every request is served: the system "faces challenges due to highly overloaded scenarios". Rejecting early, before a request consumes prefill capacity it will not get to use, is a first-class design element rather than an error path. Most reference architectures drawn by vendors omit it.

The router is a cache index, not a load balancer

In llm-d, a kvblock.Index maps block hashes to pods and a kvblock.Scorer "ranks each pod based on consecutive matching blocks from the start of the prompt". Least-connections routing actively destroys value here: it spreads requests that should have been concentrated.

The pools are sized asymmetrically

DeepSeek runs prefill at "Routed Expert EP32, MLA/Shared Expert DP32" across 4 nodes and decode at "Routed Expert EP144, MLA/Shared Expert DP144" across 18 nodes. The ratio, not the split, is the design: decode is where the tokens and the time go.

The KV cache lives on borrowed memory

Mooncake "leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster". Meta tiers it deliberately: system prompts in HBM, active chat history in DRAM, "chat history from less engaging users… offloaded to flash". You already paid for this memory when you bought the GPU nodes.

Transfer is the contested seam

vLLM chose a store: "we now go for KVCache-store-based design. If you prefer direct P2P please raise concerns". Two years later an RFC argues the resulting pull model serialises decode behind prefill, and proposes pushing instead. The seam is still moving.

Prefill nodes can cheat on memory

A vLLM RFC points out that a prefill-only node never decodes, so it "can directly store the generated KV caches to CPU instead of GPU", claiming roughly 7x context length on an A100 40GB. Specialising the pool changes what the pool is allowed to do, which is the real argument for splitting.

What none of these accounts describe is how the shape behaves during a rollout. Anyscale's November 2025 write-up gets closest, and its observation is the one to carry into a design review: once you disaggregate, "engine replicas are no longer independent in optimal serving patterns", and the orchestrator "must coordinate data parallel attention, expert parallel routing, and disaggregated prefill/decode execution across potentially heterogeneous hardware". You have traded a fleet of interchangeable replicas for a distributed system with topology. Nobody has published what happens when you deploy to it on a Friday.

The decisions that matter

Four forks, each with the chosen path, the rejected one, and the condition that flips it.

Decision: split prefill and decode onto separate pools, or keep one pool and interleave?

Chosen

Separate pools, by DeepSeek, Moonshot AI, Meta and Perplexity
Each phase gets its own batch size, parallelism and hardware
Prefill stops stalling in-flight decodes, so tail inter-token latency becomes controllable

Rejected

One pool with chunked prefill, as in Sarathi-Serve
Rejected at large scale because the two phases cannot be tuned independently
But it measured 2.6x to 5.6x higher serving capacity under SLO on single-node deployments

Flips when

You are on one node, or a handful. Chunked prefill wins outright and costs nothing operationally.
TTFT is your binding SLO: disaggregation adds 50–100ms of it at Meta's measurement.
You need throughput. vLLM's own documentation states "disaggregated prefill DOES NOT improve throughput".

That last point deserves its own paragraph, because it contradicts most of what is written about disaggregation. The vLLM project documents the feature it built with a sentence in capitals saying it does not improve throughput, and describes its purpose as "tuning time-to-first-token (TTFT) and inter-token-latency (ITL) separately" and controlling "tail ITL". Disaggregation is a latency-shaping tool that lets you buy different hardware for different phases. If you adopt it expecting more tokens per second per GPU, the project that wrote it has told you in advance that you will be disappointed.

Decision: route by cache locality, or by load?

Chosen

Prefix-cache-aware routing, in llm-d, in GKE Inference Gateway and in AIBrix
llm-d's own benchmark: P90 TTFT of 0.542s precise, against 92.551s random
Red Hat reports 87.4% cache hit rate with 99.92% of traffic on one pod

Rejected

Least-connections or round-robin, the default in every ingress you already run
Loses because it spreads shared prefixes across replicas, so each one recomputes
llm-d measures load-aware routing at 4,428.7 tok/s against 8,730 for precise

Flips when

Your prompts do not share prefixes. llm-d names RAG explicitly: "the exact documents and their order often change between queries, breaking simple prefix patterns".
Concentrating 99.92% of traffic on one pod is a hot-spot risk, not only a cache win.
Every published measurement of this was produced by the party that built the router.

Decision: move KV cache through a store, or point to point between replicas?

Chosen

vLLM, December 2024: "we now go for KVCache-store-based design"
Keyed on the prefix-cache hash, so transfer and reuse share one addressing scheme
Serves a second use case: long documents whose KV will not fit in GPU or CPU memory

Rejected

Direct peer-to-peer transfer, deferred with an invitation to object in Slack
Simpler per hop, but gives no reuse across requests and no place to spill

Flips when

The store's pull model becomes the bottleneck. RFC 36923, March 2026: "D must wait for P to finish before it can begin transferring KV data, adding latency to the critical path."
Its authors measure P99 TTFT falling from 1,125.98ms to 302.37ms by pushing instead of pulling.

Decision: shed load on arrival, or queue and hope?

Chosen

Mooncake rejects early, and predicts rather than measures when deciding
Under a 2x replay of 23,000 real traces, prediction-based rejection dropped 3,589 requests against the baseline's 4,183

Rejected

Rejecting on currently observed load, the obvious implementation
Loses because of a delay between the prefill decision and the decode consequence

Flips when

Never, at scale. The naive version is actively harmful: it "causes fluctuations and phase staggering between the loads on prefill and decoding instances".
Below the point where you are ever overloaded, skip the machinery entirely.

Figure 3 · Which of these you actually need

Read this as a sequence of tests, not a menu: each step costs operational complexity, and the published evidence only justifies the later steps at scale. Sources: Sarathi-Serve, vLLM docs, llm-d.

Diagram source

Decision	Chosen	Rejected	Because	Evidence
Phase separation	Separate prefill and decode pools	Single pool, chunked prefill	Independent tuning of TTFT and ITL; not throughput	vLLM docs, 2026
Pool sizing	Asymmetric, decode-heavy	Equal pools	Decode holds the tokens and the time: 4 nodes prefill against 18 decode	DeepSeek, 2025
Routing	Longest-prefix block match	Least connections	Cache locality beats load spreading when prefixes are shared	Red Hat, 2025
KV transfer	Cache store keyed on prefix hash	Direct peer-to-peer	Reuse across requests and a place to spill long documents	vLLM RFC 10818, 2024
Transfer direction	Pull, today	Push, proposed	Pull serialises decode behind prefill; push measured 1.2x to 3.0x better	vLLM RFC 36923, 2026
Overload	Predictive early rejection	Rejection on observed load	Observed-load rejection oscillates the two pools out of phase	Mooncake, 2024
Speculation length	Fixed k	Per-request adaptive k	Acceptance rate is a poor control signal; goodput argued instead	vLLM PR 26504, closed 2026

The last row is the most instructive one in the table, and it is a negative result. Adapting the number of speculative tokens per request is an obviously good idea that has been proposed repeatedly and has not landed. PR 26504 implemented it, ran for eight months, and was closed as stale in June 2026. The substantive objection in the thread was not that the idea is wrong but that the signal is: a reviewer reported that the acceptance-threshold heuristic "struggled on this workload" and argued that measured goodput would be a better control input. If you are planning to build adaptive speculation yourself, that thread is the cheapest month of your life.

What broke in production

Three published incidents and two documented pathologies, grouped by failure class. Each carries the rule worth taking into a design.

Class 1 · The service is up and the answers are wrong

Postmortem

A load-balancing change routed short requests to long-context servers

AssumptionRouting a request to the wrong replica of the same model is a performance question, not a correctness one.

What happened"Some Sonnet 4 requests were misrouted to servers configured for the upcoming 1M token context window." A routine load-balancing change on 29 August "unintentionally increased the number of short-context requests routed to the 1M context servers".

Blast radius"At the worst impacted hour on August 31, 16% of Sonnet 4 requests were affected." Roughly 30% of Claude Code users in the window had at least one message degraded. Availability was normal throughout.

FixEvaluations sensitive enough to separate working from broken implementations, run continuously against production rather than in CI.

Design ruleIf two replica classes of the same model differ in configuration, the router is part of your correctness surface. Treat a routing change like a model change, not like a capacity change.

SourceAnthropic postmortem, 17 September 2025

Postmortem

A compiler bug made the model pick the wrong token, on some batch sizes only

AssumptionServing the same weights on several accelerator families is a portability problem, solved once.

What happenedA token-selection change "inadvertently triggered a latent bug in the XLA:TPU compiler". Operations "that should have agreed on the highest probability token were running at different precision levels", and the approximation "sometimes returned completely wrong results, but only for certain batch sizes and model configurations".

Blast radiusWrong-language tokens and syntax errors in output, on Opus 4.1 and Opus 4 from 25 to 28 August and Sonnet 4 from 25 August to 2 September.

FixSwitched from approximate to exact top-k and standardised additional operations on fp32.

Design ruleBatch size is a correctness variable, not only a throughput knob. Any accuracy test that runs at one batch size on one accelerator is not testing your fleet.

SourceAnthropic postmortem, 17 September 2025

Anthropic names a second-order cause that is worth more to most readers than either bug: their own privacy controls slowed the investigation. Internal rules "limit how and when engineers can access user interactions with Claude", which "protects user privacy but prevents engineers from examining the problematic interactions needed to identify or reproduce bugs". If you are designing an inference platform under a data-handling regime, that trade is now documented, with a month of degraded output as the price. Build the privacy-preserving debugging path before you need it.

Class 2 · The GPUs were fine and the control plane was not

Postmortem

A telemetry rollout took down the Kubernetes control plane, and DNS took the inference service with it

AssumptionThe data plane keeps serving if the control plane dies, so a control-plane change is low risk.

What happenedA new telemetry service made "every node in each cluster to execute resource-intensive Kubernetes API operations whose cost scaled with the size" of the cluster, which "overwhelmed the Kubernetes API servers". The data plane survived, but "DNS relies on the control plane" and services stopped being able to find each other.

Blast radiusAll OpenAI services degraded or unavailable from 3:16pm to 7:38pm PST on 11 December 2024. It passed a staging cluster cleanly: the cost of the operation scaled with cluster size, and staging was small.

FixPhased rollouts with control-plane monitoring, fault injection, a break-glass path to the control plane, and decoupling data plane from control plane dependencies.

Design ruleDNS caching is a detection delay: "DNS caching made the issue far less visible until the rollouts had begun fleet-wide". Any cache between a change and its symptom sets your minimum blast radius.

SourceOpenAI incident report, 11 December 2024

Postmortem

A blank field in a quota policy dereferenced a null pointer in every region at once

AssumptionInference endpoints fail with their own dependencies, and a quota-checking path is not one of them.

What happenedA Service Control feature shipped on 29 May "did not have appropriate error handling nor was it feature flag protected". A policy with blank fields entered the regional Spanner tables and the "metadata was replicated globally within seconds", crashing the binary everywhere.

Blast radiusDozens of products on 12 June 2025 including "Vertex AI Online Prediction". Recovery in us-central1 took "up to ~2h 40 mins" because restarting tasks "created a herd effect on the underlying infrastructure it depends on", and Service Control "did not have the appropriate randomized exponential backoff implemented".

FixFeature flags on the serving path, a red-button kill switch, and randomised backoff on restart.

Design ruleEnumerate every synchronous dependency in front of a token: quota, auth, policy, metering. Each one is a global single point of failure with a faster replication topology than your model.

SourceGoogle Cloud incident report, 12 June 2025

Class 3 · Documented pathologies with no incident report attached

Paper

Rejecting requests on observed load makes the two pools oscillate

AssumptionWhen you are overloaded, reject based on how loaded you currently are.

What happenedMooncake's authors found a delay between admitting at prefill and the consequence arriving at decode. "This delay causes fluctuations and phase staggering between the loads on prefill and decoding instances": each pool ends up idle while the other saturates.

Blast radiusNot an incident; a measured control-loop pathology. Under a 2x replay of 23,000 real traces, the baseline rejected 4,183 requests against 3,589 with prediction.

FixReject on a prediction of load after a stated interval rather than on load now.

Design ruleAny admission control across two pipelined stages needs to act on predicted downstream state. Controlling on the upstream measurement builds an oscillator.

SourceMooncake, Moonshot AI and Tsinghua, June 2024

Talk

Blast radius grows superlinearly once a request spans forty GPUs

AssumptionScaling a model across more GPUs is a capacity decision.

What happenedMeta's inference lead describes the arithmetic directly: "40 GPUs in one partition will take down your entire process group… 3% random failures for your GPU cards, the blast radius will exponentially grow."

Blast radiusStructural rather than a single event. She also reports needing "a dedicated job scheduler to allocate hosts for distributed inference" because ordinary scheduling does not express the constraint.

FixTreat the process group as the unit of failure and place it deliberately; do not let a scheduler spread it.

Design ruleAvailability of a distributed inference replica is roughly the per-GPU availability raised to the group size. Compute that number before choosing a parallelism strategy.

SourceYe (Charlotte) Qi, QCon San Francisco, May 2025

Figure 4 · How admission control on observed load builds an oscillator

Each step is locally correct and the loop is globally wrong: the signal the gate reads is one interval behind the consequence it causes, so the two pools end up saturating in antiphase. Source: Mooncake, 2024.

Diagram source

Figure 5 · The path of a silent quality failure, and where detection was not

Every arrow here succeeded: no error was raised, no SLO was breached, and the first detector in the chain was a user complaint six days later. Source: Anthropic postmortem, September 2025.

Diagram source

Read those five entries together and the shape of the risk changes. Two of the three published incidents had nothing to do with GPUs, models or serving frameworks: they were a telemetry rollout and a quota-policy code path. The Microsoft study of 156 high-severity incidents at a hyperscale inference service points the same way at a population level: about 60% were inference-engine failures with timeouts and resource exhaustion dominating, and only 4% were classified as operational. That is not a story about model serving being exotic. It is a story about model serving being an ordinary distributed system with an unusually expensive working set and an unusually quiet failure mode.

Numbers you can plan against

Every figure carries its source and its date. Where a number is a vendor claim or a self-benchmark rather than an independent measurement, the table says so.

Metric	Value	At	Context	As of	Source
Prefix cache hit rate on real traffic	56.3%	DeepSeek	342B of 608B input tokens in 24h served from on-disk KV cache	2025-02	measured
Ceiling on KV reuse	~50%	Moonshot AI	"up to only 50% of the KVCache can be reused in our current workloads"	2024-06	measured
Daily serving cost	$87,072	DeepSeek	226.75 nodes average, 8 H800 each, at $2 per GPU-hour	2025-02	measured
Theoretical cost profit margin	545%	DeepSeek	$562,027 notional revenue at R1 list price; actual revenue stated as substantially lower	2025-02	derived by DeepSeek
Cost per 1M output tokens	$0.20	SGLang on 96 H100	Independent reproduction of the DeepSeek architecture	2025-05	measured
Per-node throughput	52.3k / 22.3k tok/s	SGLang	Input / output per node, 2000-token inputs, 12 nodes of 8 H100	2025-05	measured
Output speed per user	20–22 tok/s	DeepSeek	Average across production traffic	2025-02	measured
TTFT cost of disaggregation	+50 to 100 ms	Meta	"you have to transfer hundreds of megabytes of KV cache"	2025-05	practitioner report
Latency and capacity gain from tiered KV cache	>50%	Meta	HBM / DRAM / flash tiering, both metrics	2025-05	practitioner report
P90 TTFT, precise vs random routing	0.542s vs 92.551s	llm-d	Qwen-32B, 8 pods of 2 H100, shared-prefix workload	2025-09	self-benchmark
Cache hit rate, cache-aware routing	87.4%	llm-d / Red Hat	4,176 of 4,776 queries; 99.92% of traffic to one pod	2025-10	self-benchmark
Serving capacity from chunked prefill	2.6x to 5.6x	Sarathi-Serve	Mistral-7B on one A100 through Falcon-180B, under SLO, against vLLM	2024-03	peer-reviewed
P99 TTFT, push vs pull KV transfer	1,125.98 → 302.37 ms	vLLM RFC 36923	512 input / 128 output tokens on AWS P5en; 1.2x to 3.0x across shapes	2026-03	contributor benchmark
Incident mix	~60% engine	Microsoft	156 high-severity incidents; timeouts ~40% and resource exhaustion ~29% within that class	2025-06	peer-reviewed
HTTP 408 rate after connection-liveness fix	2.72% → 0.47%	Microsoft	Normalised, following the mitigation	2025-06	measured
Serving scale and cost reduction	20k QPS, 33x	Character.AI	Cost reduction since late 2022; claims 13.5x cheaper than leading commercial APIs	2024-06	self-reported
Savings from insourcing a small model	~$1M / year	Perplexity	Related-Questions feature only, against third-party APIs; 435M queries/month overall	2024-12	vendor blog
TTFT, GKE Inference Gateway vs a third party	188.36 vs 2624.73 ms	Google Cloud	Llama 3.1 8B on 8 A100 40GB, shared-prefix workload, commissioned benchmark	2026-06	vendor claim

Read these carefully

The two rows worth the most are the two cache hit rates, because they are the only figures here from production traffic at scale that were not produced to sell something, and they agree: DeepSeek measured 56.3% and Moonshot AI reports a ceiling around 50%. Plan on roughly half your input tokens being reusable and you will be close. Every other cache figure in this table, including 87.4% and the routing comparisons, comes from a benchmark designed by the party that built the router, on a workload chosen to have shared prefixes. The DeepSeek 545% margin is arithmetic on list prices, and DeepSeek says so itself: actual revenue was "substantially lower" because much of the service is free. Treat the row as evidence that inference at scale can be gross-margin positive, not as a margin you will see. The Sarathi-Serve multipliers are measured against a 2024 vLLM baseline that has since changed substantially; the mechanism transfers, the multiplier does not.

The evidence wall

Every source behind this page, graded and dated. Filter by kind. The full claim-by-claim ledger, with the quote supporting each one, ships beside this file as sources.md.

Postmortem Anthropic2025-09

A postmortem of three recent issues

Three overlapping infrastructure bugs degraded output quality for about a month without degrading availability: a context-window routing error, a TPU server misconfiguration corrupting token generation, and an XLA:TPU miscompilation of approximate top-k. Unusually candid about why the evaluations missed it and why privacy controls slowed the fix.

Carry forwardRouting between replica classes of one model is a correctness surface, and noisy evals will not defend it.

anthropic.com/engineering/a-postmortem-of-three-recent-issues

Postmortem OpenAI2024-12

Incident report, 11 December 2024

A telemetry deployment triggered Kubernetes API operations whose cost scaled with cluster size, collapsing the control plane in the largest clusters. It passed staging because staging was small. The data plane kept running but DNS did not, so services could not find each other, and engineers were locked out of the control plane they needed to fix it.

Carry forwardTest changes whose blast radius scales with fleet size against fleet-sized clusters, and keep a break-glass path into the control plane.

status.openai.com/incidents/ctrsv3lwd797

Postmortem Google Cloud2025-06

Incident report, multiple products, 12 June 2025

An unflagged quota-policy code path hit a null pointer when a policy with blank fields replicated globally within seconds. Vertex AI Online Prediction was among dozens of affected products. Recovery in us-central1 took about 2h40m because restarts stampeded the Spanner table they depend on, with no randomised backoff.

Carry forwardEvery synchronous dependency in front of a token is a global failure domain; check each for feature flags, a kill switch and backoff on restart.

status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW

Paper Microsoft2025-10

Enhancing reliability in AI inference services: an empirical study on real production incidents

A provider-internal taxonomy validated on 156 high-severity incidents from April to June 2025. About 60% were inference-engine failures, dominated by timeouts and resource exhaustion; 74% were auto-detected but 48% of mitigations were monitor-only. Connection liveness cut the normalised HTTP 408 rate from 2.72% to 0.47%.

Carry forwardBudget your reliability work against the measured distribution: timeouts on long streaming responses, not exotic GPU faults.

arxiv.org/abs/2511.07424

Paper Moonshot AI / Tsinghua2024-06

Mooncake: a KVCache-centric disaggregated architecture for LLM serving

The serving platform behind Kimi. Separates prefill and decode clusters and builds a KV pool from the CPU, DRAM and SSD already present in the GPU cluster. The section on overload is the rare published treatment of admission control, including the oscillation that naive early rejection produces.

Carry forwardAdmission control across two pipelined pools must act on predicted downstream load, or it becomes an oscillator.

arxiv.org/html/2407.00079v1

Paper Microsoft Research / Georgia Tech2024-03

Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve

The strongest argument for not disaggregating. Chunked prefills split a prefill into near-equal chunks and schedule them without pausing running decodes, measuring 2.6x higher serving capacity for Mistral-7B on one A100 and up to 5.6x for Falcon-180B, under latency SLOs, against the vLLM of the day.

Carry forwardExhaust single-pool scheduling before buying a second pool; the capacity is there and it costs no orchestration.

arxiv.org/abs/2403.02310

Case study DeepSeek2025-02

DeepSeek-V3/R1 inference system overview

The most specific public account of a large inference deployment: prefill at EP32 over four nodes, decode at EP144 over eighteen, 226.75 nodes average occupancy at $2 per GPU-hour, 608B input and 168B output tokens in a day, and 56.3% of input tokens served from the on-disk KV cache.

Carry forwardSize decode far larger than prefill, and expect roughly half your input tokens to be cache hits on conversational traffic.

github.com/deepseek-ai/open-infra-index

Talk Meta2025-05

Ye (Charlotte) Qi, Scaling large language model serving infrastructure at Meta, QCon SF

The single most useful practitioner account in this set, and it has a transcript. Tiered KV caching across HBM, DRAM and flash for over 50% reduction in latency and capacity; a measured 50 to 100 ms TTFT cost for disaggregation; blast radius arithmetic at forty GPUs per process group; and the admission that no good autoscaling signal exists.

Carry forward"QPS obviously does not work. Tokens per second works under a lot of caveats." Pick your scaling signal deliberately and instrument for it.

infoq.com/presentations/llm-meta

Eng blog LMSYS / SGLang2025-05

Deploying DeepSeek with PD disaggregation and large-scale expert parallelism on 96 H100 GPUs

An independent reproduction of DeepSeek's published architecture on twelve nodes, reaching 52.3k input and 22.3k output tokens per second per node at about $0.20 per million output tokens. Names its own limits honestly, including that expert parallelism "often leads to uneven workload distribution across GPUs".

Carry forwardExpert-parallel load imbalance worsens with GPU count, and the balancer needs production traffic to be tuned against.

lmsys.org/blog/2025-05-05-large-scale-ep

Eng blog llm-d (IBM, Red Hat, Google, Alibaba)2025-09

KV-cache wins you can see

A four-way comparison of routing strategies on a shared-prefix workload: P90 TTFT of 0.542s for precise prefix-cache scheduling against 92.551s for random, with throughput of 8,730 against 4,428.7 tokens per second for load-aware. Notable for saying where it does not apply.

Carry forwardPrefix routing collapses for RAG, where "the exact documents and their order often change between queries".

llm-d.ai/blog/kvcache-wins-you-can-see

Eng blog Red Hat / IBM2025-10

Master KV cache aware routing with llm-d

The implementation detail behind the benchmark: a block index mapping hashes to pods, a prefix store to avoid re-tokenising, and a scorer that ranks pods by consecutive matching blocks from the start of the prompt. Reports 87.4% hit rate and 99.92% of traffic landing on one pod.

Carry forwardCache-aware routing deliberately creates hot spots; pair it with a load scorer or you have built a single point of contention.

developers.redhat.com

Eng blog Anyscale2025-11

Ray Serve LLM: wide-EP and disaggregated serving with vLLM

The clearest published statement of what disaggregation costs operationally: replicas stop being independent, and something has to coordinate data-parallel attention, expert routing and the prefill/decode split across possibly heterogeneous hardware. Reports 2.4k tokens per second per H200 on Nebius with InfiniBand.

Carry forwardDisaggregation converts a stateless replica fleet into a topology-aware placement problem. Budget for that, not just for GPUs.

anyscale.com/blog

Eng blog Character.AI2024-06

Optimizing AI inference at Character.AI

Serving economics at conversational scale: around 20,000 queries per second, serving costs reduced by at least 33x since late 2022, and a claim of 13.5x cheaper than the most efficient leading commercial APIs. Light on mechanism, unusually specific on outcome.

Carry forwardThe order-of-magnitude savings in this space come from attention and cache design, not from buying cheaper GPUs.

blog.character.ai

Decision record vLLM project2024-12

RFC 10818: disaggregated prefilling and KV cache transfer roadmap

The decision to build KV transfer around a cache store rather than direct peer-to-peer links, stated in one line with an open invitation to object. Also the roadmap for XpYd topologies, asynchronous layer-by-layer transfer and third-party store integrations.

Carry forwardThe store-versus-P2P choice is what makes cross-request reuse possible; it is not just a transport detail.

github.com/vllm-project/vllm/issues/10818

Decision record vLLM project2024-06

RFC 5557: implement disaggregated prefilling via KV cache transfer

The original design thread, six months earlier. Proposes a communicator and a KV database keyed on the automatic-prefix-caching hash, and names long-document reuse as a co-equal motivation alongside disaggregation itself.

Carry forwardPrefix-cache hashing is the shared addressing scheme underneath both reuse and transfer; design it once.

github.com/vllm-project/vllm/issues/5557

Source vLLM project2026-03

RFC 36923: KV push from prefill to decode

An argument that the shipped pull model serialises decode behind prefill, with a push-based alternative measured at 1.2x to 3.0x better and a P99 TTFT falling from 1,125.98 ms to 302.37 ms on one shape. Also honest about what the proposal does not handle: multi-node, pipelining, and failure fallback.

Carry forwardIf you are running disaggregation today, the transfer direction is a live performance bug, not a settled design.

github.com/vllm-project/vllm/issues/36923

Source vLLM project2025-10

PR 26504: adaptive speculative decoding, closed as stale

Eight months open, then closed. The interesting part is the review: acceptance rate as a control signal "struggled on this workload", with measured goodput argued as the better input. A worked example of a good idea failing on its choice of feedback variable.

Carry forwardBefore building adaptive speculation, decide whether you are controlling on acceptance rate or on goodput. The first has been tried.

github.com/vllm-project/vllm/pull/26504

Source vLLM project2025-06

RFC 19038: prefill-only optimisations for PD disaggregation

Once a node only ever prefills, it can put generated KV in CPU memory instead of GPU memory, claimed at roughly 7x context length on an A100 40GB. The specialisation argument for disaggregation, stated more concretely than in any vendor material.

Carry forwardThe real return on splitting pools is what each pool is then allowed to do differently, not the split itself.

github.com/vllm-project/vllm/issues/19038

Paper AIBrix Team2025-02

AIBrix: towards scalable, cost-effective large language model inference infrastructure

A cloud-native serving framework describing the same five parts as everyone else, but naming routing as "prefix-aware, load-aware" in one scorer rather than treating cache locality and load as competing strategies. Claims a "50% increase in throughput" and a "70% reduction in inference latency".

Carry forwardCache locality and load are inputs to one scoring function, not a choice between two routers; the numbers here are the project's own.

arxiv.org/abs/2504.03648

Vendor vLLM project2026-08

Disaggregated prefilling (experimental), official documentation

Documentation that argues against its own feature's most common selling point: "disaggregated prefill DOES NOT improve throughput". Its stated purpose is tuning TTFT and ITL separately and controlling tail ITL. Still marked experimental.

Carry forwardQuote this line in the design review where someone proposes disaggregation as a throughput project.

docs.vllm.ai/en/latest/features/disagg_prefill

Vendor NVIDIA / Perplexity2024-12

Perplexity AI serves 400 million search queries a month using the NVIDIA inference stack

Vendor material, so read it for shape rather than for outcomes. Useful anyway: 435 million queries a month across more than twenty models, an in-house front-end scheduler routing on load against SLAs, and about $1M a year saved by insourcing one small-model feature.

Carry forwardThe published savings come from moving small, high-volume features off third-party APIs, not from self-hosting the frontier model.

developer.nvidia.com

Vendor Google Cloud2026-06

GKE Inference Gateway prefix caching accelerates AI inference

A commissioned benchmark on Llama 3.1 8B across eight A100s, reporting 92.8% lower mean TTFT and 15.7% more output token throughput against an unnamed third party on a shared-prefix workload. No caveats offered about workloads without shared prefixes.

Carry forwardEvery number here depends on the prefix-sharing ratio of the test workload; ask for that ratio before believing any of it.

cloud.google.com/blog

Where the record runs out

Three gaps are worth naming because they are where your risk is. First, no public postmortem describes a prefill/decode split failing in production. There are design documents, benchmarks and vendor pages in quantity, and no incident report; the closest published material is Anyscale's warning that replicas stop being independent and Meta's blast-radius arithmetic. Second, prefix-cache hit rates on real traffic have been published by exactly two organisations, DeepSeek and Moonshot AI, and every other hit-rate figure available is a benchmark by a party selling the router. Third, the conference-talk layer on this topic is large and almost entirely locked in video: two attempts to retrieve talk pages for this guide returned a 403 and an empty shell, so one talk with a published transcript is cited here and the rest were left out rather than cited from summaries.

Build a miniature, then productionise it

Six rungs. The first three run on one GPU or a laptop; the last three are where the arguments in this guide stop being abstract.

Measure the two phases separately

Serve any small open-weight model and record TTFT and inter-token latency independently across a range of input lengths at fixed concurrency. Plot them against each other.

Done when: you can point at the input length where TTFT starts dominating total latency for your prompt shape. Teaches: why one number for "latency" is useless here, and which phase your workload actually lives in.

Break the decode stream with a prefill

Hold a steady stream of short generations, then inject one very long prompt. Watch the inter-token latency of the in-flight requests. Then enable chunked prefill and repeat.

Done when: you have a chart showing the ITL spike and its disappearance. Teaches: the interference that Sarathi-Serve removes and that disaggregation removes differently, at different cost.

Prove your own prefix-reuse ceiling

Take a day of your real prompts, hash them into blocks the way automatic prefix caching does, and compute what share of input tokens would be cache hits under perfect routing. Compare against DeepSeek's 56.3% and Mooncake's 50% ceiling.

Done when: you have a single number for your traffic. Teaches: whether cache-aware routing is worth anything to you before you deploy a router to find out.

Route two replicas by prefix, and then break it

Put a scorer in front of two replicas that ranks by longest consecutive block match. Measure TTFT. Then replay a RAG-shaped workload where document order varies, and measure again.

Done when: you have reproduced both the win and the collapse llm-d describes. Teaches: that the routing strategy is a bet on your prompt distribution, not a general improvement.

Disaggregate, and pay the transfer

Split into a prefill instance and a decode instance with KV transfer between them. Measure the TTFT delta against your single pool, and the bytes moved per request.

Done when: your measured TTFT penalty is within sight of Meta's 50 to 100 ms and you can state the bytes per request. Teaches: what you are buying, and that vLLM's documentation was right about throughput.

Build the detector that would have caught Anthropic's bug

Run a fixed golden set continuously against production, not CI, tagged by replica class and batch size, alerting on distribution shift rather than on a pass/fail threshold. Then deliberately misroute 15% of traffic to a differently configured replica and see whether it fires.

Done when: the detector fires on a 15% misroute within an hour. Teaches: the only defence against the failure class in section 04, and why "we have evals" is not it.

Keep hunting

The queries that actually surfaced the material above. This page will go stale; these will not.

Design arguments, not documentation

vLLM RFC prefill decode disaggregation design github issue
site:github.com/vllm-project "[RFC]" KV cache transfer
vllm closed pull request "closing as stale" scheduler OR speculative

Incidents and degradation

LLM inference postmortem "root cause" degraded quality routing
status.<provider>.com incident inference "what happened"
"inference" incident taxonomy production empirical study arxiv

Numbers from real traffic

"KV cache" hit rate production "we" tokens per day cost
inference system overview cost profit margin GPU hour nodes
"cost per 1M output tokens" H100 OR H800 measured benchmark

Practitioner accounts with transcripts

site:infoq.com presentations LLM serving infrastructure scaling
QCon OR "Ray Summit" LLM inference talk transcript KV cache
"we replaced" OR "we moved off" inference server production

References

Anthropic, A postmortem of three recent issues Anthropic Engineering, 17 September 2025. Checked 2026-08-28.
OpenAI, Incident report for 11 December 2024 OpenAI status, December 2024. Checked 2026-08-28.
Google Cloud, Multiple products incident report, 12 June 2025 Google Cloud Service Health, June 2025. Checked 2026-08-28.
Ranganathan, Zhang and Wu, Enhancing reliability in AI inference services: an empirical study on real production incidents Microsoft, arXiv:2511.07424, 17 October 2025. Checked 2026-08-28.
Qin et al., Mooncake: a KVCache-centric disaggregated architecture for LLM serving Moonshot AI and Tsinghua University, arXiv:2407.00079, 24 June 2024. Checked 2026-08-28.
Agrawal et al., Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve Microsoft Research and Georgia Tech, arXiv:2403.02310, 4 March 2024. Checked 2026-08-28.
DeepSeek, DeepSeek-V3/R1 inference system overview open-infra-index, February 2025. Checked 2026-08-28.
Ye (Charlotte) Qi, Scaling large language model serving infrastructure at Meta QCon San Francisco, presented 2024, transcript published 29 May 2025. Checked 2026-08-28.
The SGLang Team, Deploying DeepSeek with PD disaggregation and large-scale expert parallelism on 96 H100 GPUs LMSYS Org, 5 May 2025. Checked 2026-08-28.
Ayoub, Harnik, Smith, Swain, Wang, Yin and Yan, KV-cache wins you can see llm-d, 24 September 2025. Checked 2026-08-28.
Nuland and Ayoub, Master KV cache aware routing with llm-d for efficient AI inference Red Hat Developer, 7 October 2025. Checked 2026-08-28.
Eicher, Hakhamaneshi and Qiao, Ray Serve LLM on Anyscale: wide-EP and disaggregated serving with vLLM Anyscale, 26 November 2025. Checked 2026-08-28.
Character.AI, Optimizing AI inference at Character.AI Character.AI blog, 20 June 2024. Checked 2026-08-28.
vLLM, RFC: disaggregated prefilling and KV cache transfer roadmap, issue 10818 Opened by KuntaiDu, 2 December 2024. Checked 2026-08-28.
vLLM, RFC: implement disaggregated prefilling via KV cache transfer, issue 5557 Opened by KuntaiDu, 14 June 2024. Checked 2026-08-28.
vLLM, RFC: KV push from prefill to decode node using NIXL connector, issue 36923 Opened by snadampal, 12 March 2026. Checked 2026-08-28.
vLLM, RFC: prefill-only optimizations for PD disaggregation, issue 19038 Opened by KuntaiDu, 2 June 2025. Checked 2026-08-28.
vLLM, PR 26504: add DynamicProposer for per-sequence dynamic speculative decoding Opened 9 October 2025, closed as stale 24 June 2026. Checked 2026-08-28.
The AIBrix Team, AIBrix: towards scalable, cost-effective large language model inference infrastructure arXiv:2504.03648, 22 February 2025. Checked 2026-08-28.
vLLM, Disaggregated prefilling (experimental) Official documentation. Checked 2026-08-28.
NVIDIA, Spotlight: Perplexity AI serves 400 million search queries a month using the NVIDIA inference stack NVIDIA Technical Blog, 5 December 2024. Checked 2026-08-28.
Tian and Wu, GKE Inference Gateway prefix caching accelerates AI inference Google Cloud Blog, 10 June 2026. Checked 2026-08-28.