Evidence ledger
23 sources
Checked 08 Sep 2026
Evidence ledger
One row per claim in When the model quietly gets worse: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic: how LLM serving operators detect, attribute and reverse silent output-quality regressions, the ones introduced by infrastructure and product-layer changes rather than by a new model, which leave availability, latency and error-rate dashboards untouched.
Checked date for every row: 2026-09-08.
Retrieval method note. This guide was built in a sandboxed environment whose egress policy
allows direct fetching only of github.com. Rows marked fetched were retrieved in full.
Rows marked search-verified are sources on blocked hosts whose exact quoted sentences and
figures were retrieved verbatim in this session through live web search result content, not
from memory; the URLs come from those live results. No quote in this ledger is reconstructed
from memory. The page's link checker will report the blocked hosts as unreachable from this
environment; that is a property of the build sandbox, not of the links.
| # | Org | Title | Tier | Published | Method | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Anthropic | A postmortem of three recent issues | postmortem | 2025-09-17 | search-verified | https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues | Three overlapping infrastructure bugs degraded Claude output quality Aug–early-Sep 2025; none was a model change | Postmortem covers a context-window routing error (from Aug 5, worsened by an Aug 29 load-balancer change), an output-corruption misconfiguration (Aug 25–28), and an approximate top-k miscompilation (from Aug 25) |
| 2 | Anthropic | A postmortem of three recent issues | postmortem | 2025-09-17 | search-verified | https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues | Peak blast radius of the routing bug, and its platform spread | "16% of Sonnet 4 requests" degraded at peak on Aug 31; "approximately 30% of Claude Code users" had at least one misrouted message; Bedrock peak 0.18% of Sonnet 4 requests; Vertex AI under 0.0004% |
| 3 | Anthropic | A postmortem of three recent issues | postmortem | 2025-09-17 | search-verified | https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues | The token-selection bug was a mixed-precision disagreement inside a performance optimization | Approximate top-k "sometimes returned completely wrong results, but only for certain batch sizes and model configurations"; sampling computed in bf16 while TPU vector processors optimized intermediates to fp32, and "the two disagree about which token has the highest probability" |
| 4 | Anthropic | A postmortem of three recent issues | postmortem | 2025-09-17 | search-verified | https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues | The fix traded performance back for correctness, and evals moved onto production systems | Remediation: switch from approximate to exact top-k, standardize on fp32 for sampling; process changes: more sensitive evals that "clearly separate 'working' from 'broken'", quality evals run "continuously on true production systems", faster triage of community reports without weakening privacy controls |
| 5 | Anthropic | A postmortem of three recent issues | postmortem | 2025-09-17 | search-verified | https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues | The demand-throttling theory users held was denied in writing | "We never reduce model quality due to demand, time of day, or server load." |
| 6 | Anthropic | An update on recent Claude Code quality reports | postmortem | 2026-04-23 | search-verified | https://www.anthropic.com/engineering/april-23-postmortem | A second, product-layer instance: three harness changes degraded Claude Code for about a month, again with no model change | Causes: Opus 4.6 default reasoning effort set high (reverted Apr 7); a Mar 26 change to clear old thinking blocks from idle sessions fired "every turn for the rest of the session instead of just once" (fixed Apr 10); an Apr 16 verbosity-reduction system-prompt instruction "hurt coding quality" (reverted Apr 20) |
| 7 | Anthropic | An update on recent Claude Code quality reports | postmortem | 2026-04-23 | search-verified | https://www.anthropic.com/engineering/april-23-postmortem | The deliberate-degradation theory was denied a second time, and the serving layer cleared first | "We never intentionally degrade our models, and we were able to immediately confirm that our API and inference layer were unaffected." Usage limits were reset for all subscribers on Apr 23 |
| 8 | OpenAI | Unexpected responses from ChatGPT (incident report) | postmortem | 2024-02-21 | search-verified | https://status.openai.com/incidents/ssg8fh7sfyz3 | The first widely-seen instance of the class: a serving optimization corrupted token selection while the service stayed up | "On February 20, 2024, an optimization to the user experience introduced a bug with how the model processes language"; "the model chose slightly wrong numbers, which produced word sequences that made no sense"; "inference kernels produced incorrect results when used in certain GPU configurations" |
| 9 | OpenAI | Sycophancy in GPT-4o: what happened and what we're doing about it | postmortem | 2025-04-29 | search-verified | https://openai.com/index/sycophancy-in-gpt-4o/ | A behavioral quality regression shipped through green evals and positive A/B tests, and was fully rolled back in about four days | The Apr 25 GPT-4o update was rolled back starting the night of Apr 28, complete for free users Apr 29 |
| 10 | OpenAI | Expanding on what we missed with sycophancy | postmortem | 2025-05-02 | search-verified | https://openai.com/index/expanding-on-sycophancy/ | The only signal that fired was human judgement, and it was overridden | "Some expert testers had indicated that the model behavior 'felt' slightly off"; OpenAI launched anyway "due to the positive signals from the users who tried out the model. Unfortunately, this was the wrong call." |
| 11 | OpenAI | Expanding on what we missed with sycophancy | postmortem | 2025-05-02 | search-verified | https://openai.com/index/expanding-on-sycophancy/ | Why the gates were green: no deployment eval measured the failing property | Offline evals "weren't broad or deep enough to catch sycophantic behavior", A/B tests lacked "the right signals"; sycophancy was not tracked by any specific deployment evaluation; process change: treat behavior issues as launch-blocking and weight qualitative "vibe checks" |
| 12 | Anthropic community | claude-code issue #19468: Systematic Model Degradation and Silent Downgrading | source | 2026-01-20 | fetched | https://github.com/anthropics/claude-code/issues/19468 | What detection-by-user-report looks like from the inside: certain, angry, and mis-attributed to deliberate throttling | Users allege premium subscribers are "served inferior models without consent"; Anthropic's reply points at the structural detection gap: "Our internal privacy and security controls limit how and when engineers can access user interactions... prevents engineers from examining the problematic interactions needed to identify or reproduce bugs" |
| 13 | vLLM | PR #24583: [Proof of Concept] Made vllm deterministic (Chillee) | source | opened 2025-09-10, closed 2025-11-16 | fetched | https://github.com/vllm-project/vllm/pull/24583 | The determinism PoC was closed unmerged and superseded by first-class support | Closed by a maintainer with the note that "wider support has been merged to the main branch" with official documentation; reviewers had flagged runtime environment modification and undocumented dependencies |
| 14 | vLLM | Issue #39096: Batch invariance breaks with torch.compile and/or CUDA graphs on SM<90 | source | 2026-04-06, open | fetched | https://github.com/vllm-project/vllm/issues/39096 | Determinism is hardware- and compiler-conditional, not a solved property | "torch.compile alone is sufficient to break batch invariance on SM89"; greedy outputs diverge at token positions 32, 33, 80 with identical context; workaround is enforce_eager below SM90 and moving tests to H100 runners |
| 15 | vLLM | Issue #32992: Batch invariance fails on NVIDIA B200 with torch.compile | source | n.d. (closed; checked 2026-09-08) | fetched | https://github.com/vllm-project/vllm/issues/32992 | Same property broke again on the next GPU generation | "Batch invariance mode (VLLM_BATCH_INVARIANT=1) fails on B200 (Blackwell) but works correctly on H200 (Hopper)"; logprob differences of 0.25–0.37 under torch.compile, tied to Inductor codegen on SM100 |
| 16 | EleutherAI | lm-evaluation-harness issue #2851: Significantly different results with different backends | source | 2025-03-27, open | fetched | https://github.com/EleutherAI/lm-evaluation-harness/issues/2851 | The same weights and the same harness produce wildly different scores per engine, and the issue sits unresolved | leaderboard_ifeval on Qwen2.5-72B-Instruct GPTQ-8bit: prompt_level_strict_acc 0.50 (HF) vs 0.10 (vLLM), inst_level_loose_acc 0.8333 vs 0.2222; no maintainer root cause recorded |
| 17 | EleutherAI | lm-evaluation-harness issue #3154: repetition_penalty is incorrectly set when using vllm | source | 2025-07-17 | fetched | https://github.com/EleutherAI/lm-evaluation-harness/issues/3154 | One silently dropped sampling parameter moved a benchmark by 17 points | With Qwen2.5-Coder-7B-Instruct, gsm8k_cot scored 34.38 via vLLM vs 51.56 via HF; manually restoring repetition_penalty: 1.1 recovered 50.78; "vllm will skip all the penalties when processing the logits" |
| 18 | Moonshot AI | K2-Vendor-Verifier (repository) | source | 2025 (launched; README current) | fetched | https://github.com/MoonshotAI/K2-Vendor-Verifier | The model vendor now ships a conformance suite to police its own hosts | 4,000 requests per provider, measuring finish-reason alignment, schema accuracy and similarity to the official API; "significant differences in the toolcall performance of various open-source solutions and vendors"; official endpoint 100% schema accuracy vs 83.05% (Chutes, K2-Thinking) and 84.47% (Nebius, K2-0905) at the low end |
| 19 | zai-org | GLM-4.5 issue #53: GLM-4.5-Air vllm and llama.cpp, potential quality issues | source | 2025-08-16, closed not planned | fetched | https://github.com/zai-org/GLM-4.5/issues/53 | Quality loss across engines is reported even without quantization, and such reports often die unresolved | "Output quality, in particular the generated reasoning content, seems degraded on llama.cpp and vllm compared to the GLM-4.5-Air deployments on z.ai/bigmodel.cn"; reproduced with an unquantized variant; closed as not planned |
| 20 | vLLM | RFC issue #40628: Batch Invariance Dispatching in vLLM IR | adr | 2026-04-22, closed not planned | fetched | https://github.com/vllm-project/vllm/issues/40628 | The recorded design argument about who owns determinism in the stack | "should vLLM IR own batch-invariant dispatching as a first-class concern, or should it be handled outside the IR?"; author argues the IR should stay "unaware of batch invariance as a concept", routing via platform kernel-priority lists |
| 21 | vLLM | vllm-omni RFC issue #4864: Deterministic Rollout Support | adr | 2026-07-03 | fetched | https://github.com/vllm-project/vllm-omni/issues/4864 | RL training is the second customer of determinism, with a bitwise requirement | "verl-omni's deterministic training requires deterministic rollout output: same logical inputs -> same tokens/images, independent of batch composition and scheduling races" |
| 22 | OpenAI | Verifying gpt-oss implementations (cookbook) | vendor | 2025-08 | fetched | https://github.com/openai/openai-cookbook/blob/main/articles/gpt-oss/verifying-implementations.md | The model vendor's own verification procedure for third-party hosts, and its acceptance test | "If the model does not receive the prompts in the right format this can have cascading generation issues and at minimum a worse function calling performance"; smoke test expects "0 invalid requests and over 90% on both pass@k and pass^k"; full validation runs AIME25 x16, GPQA x8, HealthBench x1; "If you are getting similar benchmark results as those published by us and your function calling tests above succeeded you likely have a correct implementation" |
| 23 | Thinking Machines Lab | Defeating Nondeterminism in LLM Inference (Horace He) | blog | 2025-09-10 | search-verified | https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ | The folk explanation for irreproducible outputs was wrong; the real cause is batch-size-dependent kernels | Output depends on batch size because matmul, RMSNorm and attention kernels are not batch-invariant; serving load changes batch composition, so identical requests produce different outputs; ships batch-invariant kernel library plus a vLLM demo |
| 24 | LMSYS / SGLang | Towards Deterministic Inference in SGLang and Reproducible RL Training | blog | 2025-09-22 | search-verified | https://www.lmsys.org/blog/2025-09-22-sglang-deterministic/ | The measured price of determinism in a production engine | Slowdowns mostly "25% to 45%", average 34.35% across FlashInfer and FlashAttention 3 backends, vs 61.5% for the original batch-invariant-ops integration; majority of overhead from unoptimized batch-invariant matmul and attention kernels; goal to get under 20% |
| 25 | vLLM | Chasing 100% Accuracy: Debugging Kimi K2's Tool-Calling on vLLM | blog | 2025-10-28 | search-verified | https://blog.vllm.ai/2025/10/28/Kimi-K2-Accuracy.html | A correct engine plus a correct model can still ship a broken combination; conformance testing caught it | Initial tool-calling on vLLM: 218 successful parses out of over 1,200 potential tool calls, under 20% success; three chat-template compatibility issues found via K2-Vendor-Verifier; fixes brought a more than 4x improvement, "in line with expectations" |
| 26 | Simon Willison | Open weight LLMs exhibit inconsistent performance across providers | blog | 2025-08-15 | search-verified | https://simonwillison.net/2025/Aug/15/inconsistent-performance/ | Same weights, different hosts, different scores, and fixes move the numbers | Reporting Artificial Analysis's gpt-oss-120b provider benchmarks; after implementation fixes "Groq and Azure improved their scores to 93.3%" on AIME25 while "Google Vertex is new to the chart at 83.3%"; notes OpenAI added a compatibility test to the gpt-oss repo for providers |
| 27 | Simon Willison | Anthropic: a postmortem of three recent issues (commentary) | blog | 2025-09-17 | search-verified | https://simonwillison.net/2025/Sep/17/anthropic-postmortem/ | Independent practitioner reading of the postmortem, same day, confirming the detection-gap reading | Commentary republishing and analyzing the three-bug postmortem within hours of publication |
| 28 | 16x Eval | GPT-OSS Provider Evaluation: Do All Providers Perform the Same? | blog | 2025-08 | search-verified | https://eval.16x.engineer/blog/gpt-oss-provider-performance-differences | Independent small-scale replication of provider variance on identical prompts and settings | gpt-oss-120b via OpenRouter across Cerebras, Fireworks, Together, Groq: writing-task ratings 8/10 for three providers vs 7.5 on all three attempts for one; initial release had tool-calling bugs, and "the harmony tokenizer fix" changed reported scores |
| 29 | LessWrong (community) | Not Pinning Your OpenRouter Provider Might Invalidate Your Research | blog | 2026-07-23 | search-verified | https://www.lesswrong.com/posts/KsyoSAyBRXtwzSugg/not-pinning-your-openrouter-provider-might-invalidate-your | The research community measured how widespread unpinned-provider risk is | Review of influential AI-safety research codebases using OpenRouter: "31/32 (97%) of them use OpenRouter unsafely"; after a re-run changed only the provider, one paper's author conceded results "were contaminated by bad inference setups" |
| 30 | OpenRouter | Provider routing documentation | vendor | current (checked 2026-09-08) | search-verified | https://openrouter.ai/docs/guides/routing/provider-selection | The router's own remedy: quality varies by host, pin quantization when it matters | Docs acknowledge providers serve the same model at different precision and expose a quantizations field to filter; "Quantization is the hidden quality variable" (OpenRouter guidance retrieved via live search) |
| 31 | GitHub | How we evaluate AI models and LLMs for GitHub Copilot | blog | 2025-01-17 | search-verified | https://github.blog/ai-and-ml/generative-ai/how-we-evaluate-models-for-github-copilot/ | A downstream consumer treats offline evals as an integration test on every meaningful change | Over 4,000 offline tests run before production changes; evaluation re-run "whenever we make a meaningful change" to prompt, model or system logic, compared against a known baseline |
| 32 | Chen, Zaharia, Zou (Stanford/UC Berkeley) | How is ChatGPT's behavior changing over time? (arXiv 2307.09009) | paper | 2023-07 | search-verified | https://arxiv.org/abs/2307.09009 | The first systematic measurement that a hosted model's behavior drifts under the same name | GPT-4 prime-identification accuracy fell from 97.6% (March 2023 version) to 2.4% (June 2023 version); instruction-following decreased over the same window |
| 33 | Klimaszewski et al. | The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (arXiv 2605.19537) | paper | 2026-05 | search-verified | https://arxiv.org/abs/2605.19537 | The engine is an unreported experimental variable large enough to change conclusions | "the choice of backend alone can shift benchmark scores by up to 16.6 percentage points"; survey identifies 200 distinct inference engines; analysis of 35,000 ML publications finds the inference stack rarely reported; divergence driven by prefix caching, CUDA graphs, custom kernels and engine-specific logit-processing defaults |
| 34 | Sam et al. | You've Changed: Detecting Modification of Black-Box Large Language Models (arXiv 2504.12335) | paper | 2025-04 | search-verified | https://arxiv.org/pdf/2504.12335 | Detecting provider-side model changes from outside is a published, tractable technique | Method detects modification of black-box LLM APIs by comparing distributions of model outputs over time (paper retrieved via live search results; method claim from abstract) |
| 35 | (llama.cpp community study) | Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct (arXiv 2601.14277) | paper | 2026-01 | search-verified | https://arxiv.org/html/2601.14277v1 | Intrinsic metrics under-detect quality loss: equal perplexity does not mean equal capability | Quantization "schemes with similar perplexity can diverge meaningfully on instruction-following and reasoning benchmarks, so intrinsic metrics alone are insufficient to select deployment defaults" |
| 36 | InfoQ / QCon | Powering the Future: Building Your GenAI Infrastructure Stack | talk | 2026-05 | search-verified | https://www.infoq.com/presentations/infrastructure-ai-agent-development/ | Practitioner guidance: the regression suite for an LLM system must be continuously updated and trace-driven | Presentation argues for "a regression test suite that has to be continuously updated", capturing all traces where the model made decisions and evaluating each decision point against ground truth (claim taken from the published InfoQ page; timestamped citation unavailable from this build environment) |
| 37 | InfoQ | Elena Samuylova on LLM-Based Application Evaluation and LLM as a Judge (podcast) | talk | 2025-10-06 | search-verified | https://www.infoq.com/podcasts/llm-based-application-evaluation/ | The evaluation lifecycle practitioners converge on: offline regression on change, online monitoring in between | Episode covers "regression testing, monitoring, stress testing and experimental evaluations" as distinct steps, with online monitoring evaluating production behavior and regression testing gating changes (claim taken from the published InfoQ page; timestamped citation unavailable from this build environment) |
| 38 | Artificial Analysis | gpt-oss-120b: API Provider Benchmarking (provider accuracy) | casestudy | 2025-08 (launched; page current) | search-verified | https://artificialanalysis.ai/models/gpt-oss-120b/providers | Continuous third-party measurement of hosted-model accuracy exists and is methodologically serious | Provider accuracy benchmarked by running GPQA Diamond 16 times, AIME25 32 times and IFBench 8 times per provider, reporting median with min/25th/75th percentiles; 18 API providers listed for gpt-oss-120b |
Tier tally
Counting distinct documents (a document with several claims has several rows above):
- postmortem: 5 (Anthropic 2025-09-17, Anthropic 2026-04-23, OpenAI status 2024-02-21, OpenAI sycophancy 2025-04-29, OpenAI expanding 2025-05-02)
- source: 8 (claude-code #19468, vLLM PR #24583, vLLM #39096, vLLM #32992, lm-eval #2851, lm-eval #3154, K2-Vendor-Verifier, GLM-4.5 #53)
- adr: 2 (vLLM RFC #40628, vllm-omni RFC #4864)
- casestudy: 1 (Artificial Analysis provider benchmarking)
- blog: 9 (Thinking Machines, LMSYS, vLLM blog, Willison x2, 16x Eval, LessWrong, GitHub Copilot)
- paper: 4 (2307.09009, 2605.19537, 2504.12335, 2601.14277)
- talk: 2 (InfoQ presentation 2026-05, InfoQ podcast 2025-10)
- vendor: 2 (OpenAI cookbook verifying-implementations, OpenRouter provider routing docs)
Distinct documents: 33. Distinct hosts: 13 (anthropic.com, status.openai.com, openai.com, github.com, github.blog, thinkingmachines.ai, lmsys.org, blog.vllm.ai, simonwillison.net, eval.16x.engineer, lesswrong.com, openrouter.ai, arxiv.org, infoq.com, artificialanalysis.ai; 15 counting subdomains separately). Vendor share: 2/33, well under a third.
Known evidence limits, stated rather than papered over:
- Talk citations carry no timestamps: this build environment cannot fetch video or media pages, so both talk-tier claims are taken from the publisher's own page text, not from the recording.
- No provider has published the design of a production quality-canary system in any detail. The postmortems describe that continuous production evals now exist (Anthropic, row 4), not how they are built. That absence is discussed in the guide as a finding.
- No public postmortem in this corpus describes a silent quality regression that was caught by an automated gate before users reported it. Every detection story in rows 1–11 begins with user reports.