Evidence ledger
22 sources
Checked 28 Aug 2026
Evidence ledger
One row per claim in The token stream stays up and the answers get worse: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Every source behind the field guide, one row per claim. Every URL was fetched on the check date; nothing here is cited from memory. Where an artefact supports several claims it gets several rows, so a citation can never cover something its source did not say.
Tiers follow the skill's hierarchy: postmortem and source and adr are the layers ordinary
search does not surface, blog and casestudy are practitioner accounts, paper is measured
and reviewed, vendor is documentation or marketing and is capped and labelled.
Checked: 2026-08-28.
Postmortems
| # | Org | Title | Tier | Published | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|
| 1 | Anthropic | A postmortem of three recent issues | postmortem | 2025-09-17 | https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues | A routing bug can degrade output quality without degrading availability | "Some Sonnet 4 requests were misrouted to servers configured for the upcoming 1M token context window." |
| 2 | Anthropic | A postmortem of three recent issues | postmortem | 2025-09-17 | https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues | Blast radius of a load-balancing change, quantified | "At the worst impacted hour on August 31, 16% of Sonnet 4 requests were affected." and "Approximately 30% of Claude Code users who made requests during this period had at least one message routed to the wrong server type" |
| 3 | Anthropic | A postmortem of three recent issues | postmortem | 2025-09-17 | https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues | Standard evals do not catch silent quality regressions | "The evaluations we ran simply didn't capture the degradation users were reporting, in part because Claude often recovers well from isolated mistakes." and "We relied too heavily on noisy evaluations." |
| 4 | Anthropic | A postmortem of three recent issues | postmortem | 2025-09-17 | https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues | Serving the same model on several accelerator families is a correctness surface | "This change inadvertently triggered a latent bug in the XLA:TPU compiler." and "Operations that should have agreed on the highest probability token were running at different precision levels" |
| 5 | Anthropic | A postmortem of three recent issues | postmortem | 2025-09-17 | https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues | Privacy controls that block engineers from user traffic lengthen inference incidents | "This protects user privacy but prevents engineers from examining the problematic interactions needed to identify or reproduce bugs." |
| 6 | OpenAI | December 11, 2024 incident postmortem | postmortem | 2024-12-13 | https://status.openai.com/incidents/ctrsv3lwd797 | The control plane, not the GPUs, took the inference service down | The telemetry service caused "every node in each cluster to execute resource-intensive Kubernetes API operations whose cost scaled with the size" of infrastructure, which "overwhelmed the Kubernetes API servers, taking down the Kubernetes control plane in most of our large clusters." |
| 7 | OpenAI | December 11, 2024 incident postmortem | postmortem | 2024-12-13 | https://status.openai.com/incidents/ctrsv3lwd797 | Caching delayed detection until the rollout was fleet-wide | "DNS caching made the issue far less visible until the rollouts had begun fleet-wide." |
| 8 | OpenAI | December 11, 2024 incident postmortem | postmortem | 2024-12-13 | https://status.openai.com/incidents/ctrsv3lwd797 | The data plane survives without the control plane only until it needs discovery | "The Kubernetes data plane can operate largely independently of the control plane, but DNS relies on the control plane – services don't know how to contact one another" |
| 9 | OpenAI | December 11, 2024 incident postmortem | postmortem | 2024-12-13 | https://status.openai.com/incidents/ctrsv3lwd797 | Recovery was blocked by the failure itself; duration 3:16pm–7:38pm PST | "remediation was very slow because of the locked out effect" |
| 10 | Google Cloud | Incident report, multiple products, 12 June 2025 | postmortem | 2025-06-13 | https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW | A quota-policy code path took Vertex AI inference down globally | The new Service Control feature "did not have appropriate error handling nor was it feature flag protected"; affected products include "Vertex AI Online Prediction" |
| 11 | Google Cloud | Incident report, multiple products, 12 June 2025 | postmortem | 2025-06-13 | https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW | Global metadata replication turns a regional bad record into a global outage | The policy change was "inserted into the regional Spanner tables" and this "metadata was replicated globally within seconds." |
| 12 | Google Cloud | Incident report, multiple products, 12 June 2025 | postmortem | 2025-06-13 | https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW | Recovery re-broke the dependency; missing backoff extended it to ~2h40m in us-central1 | "As Service Control tasks restarted, it created a herd effect on the underlying infrastructure it depends on (i.e. that Spanner table), overloading the infrastructure. Service Control did not have the appropriate randomized exponential backoff implemented." |
Source code, issues and pull requests
| # | Org | Title | Tier | Published | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|
| 13 | vLLM project | PR #26504, eagle_dynamic adaptive speculative decoding |
source | opened 2025-10-09, closed 2026-06-24 | https://github.com/vllm-project/vllm/pull/26504 | Adaptive speculation is repeatedly proposed and repeatedly does not land | Closed by maintainer @hmellor: "Closing as stale". The PR "monitors the historical acceptance rate for each request" and "dynamically increases or decreases k". |
| 14 | vLLM project | PR #26504 review thread | source | 2025-10 to 2026-07 | https://github.com/vllm-project/vllm/pull/26504 | Acceptance rate is the wrong control signal; goodput is the argued alternative | Reviewer @KookiesNKareem noted the acceptance-threshold signal "struggled on this workload" and that measured goodput may be more effective. |
| 15 | vLLM project | RFC #36923, KV push from prefill to decode | source | 2026-03-12 | https://github.com/vllm-project/vllm/issues/36923 | Pull-based KV transfer serialises decode behind prefill | "D must wait for P to finish before it can begin transferring KV data, adding latency to the critical path." |
| 16 | vLLM project | RFC #36923, measured push-based improvement | source | 2026-03-12 | https://github.com/vllm-project/vllm/issues/36923 | Push-based transfer measurably cuts tail TTFT | "with 512 input tokens and 128 output tokens, P99 TTFT improved from 1,125.98ms to 302.37ms" on AWS P5en and Trn2; 1.2x to 3.0x across shapes. |
| 17 | vLLM project | RFC #19038, prefill-only optimisations | source | 2025-06-02 | https://github.com/vllm-project/vllm/issues/19038 | A prefill-only node can hold KV in CPU memory and buy context length | "As there is no decoding steps in prefill nodes, we can directly store the generated KV caches to CPU instead of GPU." Claimed ~7x context length on A100 40GB with DeepSeek-R1-Distill-Qwen-32B-FP8. |
Architecture decision records and design docs
| # | Org | Title | Tier | Published | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|
| 18 | vLLM project | RFC #10818, disaggregated prefilling roadmap | adr | 2024-12-02 | https://github.com/vllm-project/vllm/issues/10818 | The project chose a KV-cache-store design over direct peer-to-peer transfer | "We now go for KVCache-store-based design. If you prefer direct P2P please raise concerns in vLLM #feat-prefill-disaggregation channel" |
| 19 | vLLM project | RFC #5557, disaggregated prefilling via KV cache transfer | adr | 2024-06-14 | https://github.com/vllm-project/vllm/issues/5557 | KV transfer was designed around the prefix-cache hash from the start | The KV database uses "the hash (generated in automatic prefix caching) as the key, the corresponding KV cache tensor as the value." |
| 20 | vLLM project | RFC #5557, second use case | adr | 2024-06-14 | https://github.com/vllm-project/vllm/issues/5557 | Long-document reuse, not just disaggregation, motivated the KV store | "the GPU memory + CPU memory may not be enough to store the KV cache of all documents" |
Papers
| # | Org | Title | Tier | Published | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|
| 21 | Moonshot AI / Tsinghua | Mooncake: a KVCache-centric disaggregated architecture for LLM serving | paper | 2024-06-24 | https://arxiv.org/html/2407.00079v1 | Prefill/decode disaggregation plus a KV pool over spare CPU, DRAM and SSD | "It features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters" and "leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster" |
| 22 | Moonshot AI / Tsinghua | Mooncake, early rejection | paper | 2024-06-24 | https://arxiv.org/html/2407.00079v1 | Naive overload rejection makes prefill and decode oscillate out of phase | "This delay causes fluctuations and phase staggering between the loads on prefill and decoding instances." Fixed with "Early Rejection Based on Prediction". |
| 23 | Moonshot AI / Tsinghua | Mooncake, cache reuse ceiling | paper | 2024-06-24 | https://arxiv.org/html/2407.00079v1 | Real chat traffic has a hard ceiling on KV reuse | "up to only 50% of the KVCache can be reused in our current workloads" |
| 24 | Moonshot AI / Tsinghua | Mooncake, measured gains and rejections | paper | 2024-06-24 | https://arxiv.org/html/2407.00079v1 | Measured production gain and the residual rejection rate under 2x replay | "115% and 107% more requests" on A800 and H800 clusters; under 2x replay of 23,000 real traces, baseline rejected 4,183 requests against 3,589 with prediction-based rejection. |
| 25 | Microsoft | Enhancing reliability in AI inference services: an empirical study on real production incidents | paper | 2025-10-17 | https://arxiv.org/abs/2511.07424 | The measured failure distribution of a hyperscale inference service | 156 high-severity incidents, Apr–Jun 2025: "Inference engine failures: ~60%" dominated by timeouts (~40%) and resource exhaustion (~29%); infrastructure ~20%; model configuration ~16%; operational ~4%. |
| 26 | Microsoft | Enhancing reliability in AI inference services | paper | 2025-10-17 | https://arxiv.org/abs/2511.07424 | Most incidents are detected automatically but only half are actioned | "115 were auto-detected (≈74%)"; mitigation mix: hotfix 28%, capacity increase 10%, node rebalancing 9%, traffic routing 4%, monitor-only 48%. |
| 27 | Microsoft | Enhancing reliability in AI inference services | paper | 2025-10-17 | https://arxiv.org/abs/2511.07424 | Connection liveness is a measured, cheap mitigation for streaming timeouts | "Normalized HTTP 408 rate...declined from 2.72% to 0.47%" |
| 28 | Microsoft Research / Georgia Tech | Taming throughput-latency tradeoff in LLM inference with Sarathi-Serve | paper | 2024-03-04 | https://arxiv.org/abs/2403.02310 | Chunked prefill is the single-pool alternative to disaggregation | "Sarathi-Serve introduces chunked-prefills which splits a prefill request into near equal sized chunks and creates stall-free schedules that adds new requests in a batch without pausing ongoing decodes." |
| 29 | Microsoft Research / Georgia Tech | Sarathi-Serve, measured capacity | paper | 2024-03-04 | https://arxiv.org/abs/2403.02310 | Chunked prefill alone buys multiples of serving capacity under SLO | "For Mistral-7B on single A100 GPUs, we achieve 2.6x higher serving capacity"; "up to 3.7x" for Yi-34B on two A100s; "up to 5.6x" for Falcon-180B. |
Talks
| # | Org | Title | Tier | Published | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|
| 30 | Meta | Ye (Charlotte) Qi, Scaling large language model serving infrastructure at Meta, QCon San Francisco | talk | 2025-05-29 | https://www.infoq.com/presentations/llm-meta/ | Disaggregation is not free; it buys phase isolation with network latency | "very common to add another 50 to 100 milliseconds to your TTFT if you are doing disagg" and "you have to transfer hundreds of megabytes of KV cache" |
| 31 | Meta | Ye (Charlotte) Qi, QCon San Francisco | talk | 2025-05-29 | https://www.infoq.com/presentations/llm-meta/ | Tiered KV caching across HBM, DRAM and flash, and its measured payoff | "common system prompt might get cached in the HBM. The active user's chat history...can sit in DRAM. Chat history from less engaging users can be offloaded to flash." and "very common for us to see over 50% of reduction for both latency and capacity." |
| 32 | Meta | Ye (Charlotte) Qi, QCon San Francisco | talk | 2025-05-29 | https://www.infoq.com/presentations/llm-meta/ | Blast radius grows superlinearly with the size of a distributed inference group | "40 GPUs in one partition will take down your entire process group...3% random failures for your GPU cards, the blast radius will exponentially grow." |
| 33 | Meta | Ye (Charlotte) Qi, QCon San Francisco | talk | 2025-05-29 | https://www.infoq.com/presentations/llm-meta/ | There is no good autoscaling signal for LLM serving | "The right throughput metric is extremely tricky to find, because the bottleneck depends on your workload. QPS obviously does not work. Tokens per second works under a lot of caveats." |
| 34 | Meta | Ye (Charlotte) Qi, QCon San Francisco | talk | 2025-05-29 | https://www.infoq.com/presentations/llm-meta/ | Corroborates Anthropic: inference bugs present as plausible output, not as errors | "Inference bugs can manifest as subtle performance degradation, because LLMs are probabilistic models...you have something horribly wrong, but the result comes out still decently correct." |
Production case studies and engineering blogs
| # | Org | Title | Tier | Published | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|
| 35 | DeepSeek | DeepSeek-V3/R1 inference system overview | casestudy | 2025-02-28 | https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md | Asymmetric expert-parallel sizing between the two phases | Prefill "Routed Expert EP32, MLA/Shared Expert DP32" over 4 nodes; decode "Routed Expert EP144, MLA/Shared Expert DP144" over 18 nodes. |
| 36 | DeepSeek | DeepSeek-V3/R1 inference system overview | casestudy | 2025-02-28 | https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md | Published cost base and the theoretical margin on it | "226.75 nodes (each node contains 8 H800 GPUs)" average occupancy at "\(2 per hour" per GPU, "\)87,072" daily cost, "$562,027" theoretical daily revenue, "545%" cost profit margin. |
| 37 | DeepSeek | DeepSeek-V3/R1 inference system overview | casestudy | 2025-02-28 | https://github.com/deepseek-ai/open-infra-index/blob/main/202502OpenSourceWeek/day_6_one_more_thing_deepseekV3R1_inference_system_overview.md | The prefix cache hit rate on real traffic, at scale | 24h: input "608B, of which 342B tokens (56.3%) hit the on-disk KV cache"; output "168B"; "20–22 tokens per second" average output speed. |
| 38 | LMSYS / SGLang | Deploying DeepSeek with PD disaggregation and large-scale expert parallelism on 96 H100 GPUs | blog | 2025-05-05 | https://www.lmsys.org/blog/2025-05-05-large-scale-ep/ | An independent reproduction of the published architecture, with cost | 12 nodes × 8 H100; 52.3k input tok/s and 22.3k output tok/s per node; "$0.20 per 1M output tokens". |
| 39 | LMSYS / SGLang | Same | blog | 2025-05-05 | https://www.lmsys.org/blog/2025-05-05-large-scale-ep/ | Expert parallelism creates load imbalance that worsens with scale | "EP often leads to uneven workload distribution across GPUs"; the balancer "requires production data" and "may not reflect real-world variability". |
| 40 | llm-d (IBM, Red Hat, Google, Alibaba, DaoCloud) | KV-cache wins you can see | blog | 2025-09-24 | https://llm-d.ai/blog/kvcache-wins-you-can-see | Cache-aware routing dominates load-aware routing on a shared-prefix workload | P90 TTFT: precise 0.542s, approximate 31.083s, random 92.551s; throughput 8,730 vs 6,944.4 vs 4,428.7 tok/s. Setup: Qwen-32B on 8 pods × 2 H100. |
| 41 | llm-d | KV-cache wins you can see | blog | 2025-09-24 | https://llm-d.ai/blog/kvcache-wins-you-can-see | Prefix routing does not transfer to RAG | "While Retrieval-Augmented Generation also relies on large prefixes, reusing the KVs is more challenging. The exact documents and their order often change between queries, breaking simple prefix patterns." |
| 42 | Red Hat / IBM | Master KV cache aware routing with llm-d | blog | 2025-10-07 | https://developers.redhat.com/articles/2025/10/07/master-kv-cache-aware-routing-llm-d-efficient-ai-inference | How the router actually decides, and its measured hit rate | "A kvblock.Scorer ranks each pod based on consecutive matching blocks from the start of the prompt"; cache hit rate 87.4% (4,176 of 4,776), 99.92% of traffic to the primary pod, TTFT 340ms cached vs 2,850ms cold. |
| 43 | Anyscale | Ray Serve LLM: wide-EP and disaggregated serving with vLLM | blog | 2025-11-26 | https://www.anyscale.com/blog/ray-serve-llm-anyscale-apis-wide-ep-disaggregated-serving-vllm | Disaggregation breaks the replica as a unit of deployment | "engine replicas are no longer independent in optimal serving patterns"; the orchestrator "must coordinate data parallel attention, expert parallel routing, and disaggregated prefill/decode execution across potentially heterogeneous hardware." Measured "2.4k tps/H200 on Nebius with Infiniband". |
| 44 | Character.AI | Optimizing AI inference at Character.AI | blog | 2024-06-20 | https://blog.character.ai/optimizing-ai-inference-at-character-ai/ | Serving economics at conversational scale | "Character.AI serves around 20,000 queries per second"; "reduced our serving costs by at least 33X" since late 2022; "13.5 times less" than the most efficient leading commercial APIs. |
Vendor material (documentation and marketing, capped and labelled)
| # | Org | Title | Tier | Published | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|
| 45 | vLLM project | Disaggregated prefilling (experimental), official docs | vendor | checked 2026-08-28 | https://docs.vllm.ai/en/latest/features/disagg_prefill/ | The implementers say plainly what the feature does not do | "Disaggregated prefill DOES NOT improve throughput." It is for "Tuning time-to-first-token (TTFT) and inter-token-latency (ITL) separately" and to "control tail ITL". Marked "experimental and subject to change". |
| 46 | NVIDIA | Perplexity AI serves 400 million search queries a month using the NVIDIA inference stack | vendor | 2024-12-05 | https://developer.nvidia.com/blog/spotlight-perplexity-ai-serves-400-million-search-queries-a-month-using-nvidia-inference-stack/ | Small-model insourcing is where the published savings are | "more than 435 million queries each month"; "more than 20 AI models simultaneously"; "approximately $1 million in annual savings" on the Related-Questions feature; "a front-end scheduler built in-house that routes traffic to the appropriate pod based on their load and usage". |
| 47 | Google Cloud | GKE Inference Gateway prefix caching accelerates AI inference | vendor | 2026-06-10 | https://cloud.google.com/blog/products/containers-kubernetes/gke-inference-gateway-prefix-caching-accelerates-ai-inference | A commissioned benchmark of prefix-aware routing; treat as a claim, not a measurement | Principled Technologies benchmark, Llama 3.1 8B on 8× A100 40GB: mean TTFT 188.36 ms vs 2624.73 ms ("92.8% less TTFT"), throughput 7,169.21 vs 6,042.05 output tok/s, ITL 30.20 ms vs 81.03 ms. |
Gaps in the record
Recorded here because a named gap is more useful to an architect than a confident synthesis built on nothing.
- Conference talks are largely unverifiable by machine. Talks on this topic exist in quantity — Ray Summit, KubeCon, GTC — but the substance sits in video, and two attempts to fetch talk pages (USENIX FAST '25 and a YouTube session page) returned 403 or a shell with no content. One talk with a published transcript is cited here (row 30–34). Anything else in this layer was left out rather than cited from a summary of a summary.
- Nobody has published a failure account of disaggregated serving itself. There are design documents, benchmarks and vendor pages in quantity, and no public postmortem of a prefill/decode split failing in production. The closest published material is the operational warning in row 43 and the blast-radius arithmetic in row 32.
- Prefix-cache hit rates are published by exactly two production systems at scale (rows 23 and 37), and they agree at roughly half. Every other hit-rate figure available is a benchmark on a workload chosen by the party publishing it.
- No independent measurement of prefix-aware routing exists. Rows 40, 42 and 47 are all produced by parties who build the router being measured.
Addendum
| # | Org | Title | Tier | Published | URL | Claim taken | Supporting quote or figure |
|---|---|---|---|---|---|---|---|
| 48 | AIBrix Team | AIBrix: towards scalable, cost-effective large language model inference infrastructure | paper | 2025-02-22 | https://arxiv.org/abs/2504.03648 | Cache locality and load belong in one scoring function, not two competing routers | Components include "prefix-aware, load-aware routing", a "distributed KV cache", "LLM-specific autoscalers" and an "SLO-driven GPU optimizer"; claims "50% increase in throughput" and "70% reduction in inference latency". |