A class of production incident in which answer quality drops while every dashboard stays green, because the change was in a router, a kernel, a precision mode or a prompt, and not in the model. This guide reconstructs how the teams who have lived through it detect, attribute and reverse the failure, from their own published postmortems, issue threads and decision records.
A serving system can degrade the answers it gives without dropping a request, slowing down, or raising an error. Everything the operations team watches keeps telling them the system is healthy while the thing the system is for gets worse.
State the problem without the word "model". You take a fixed function, deploy it behind a network of routers, caches, accelerators and prompt-assembly code, and then change one of those surrounding parts to make the system faster or cheaper. The function is supposed to be untouched. It is not: the change moves the output distribution just enough that the answers get worse, and it does so without touching a single signal your monitoring is built to catch. Availability holds. Latency holds. The error rate holds. The regression is real, it is affecting a measurable fraction of traffic, and the only instrument that registers it is a human being who reads the output and says something is off.
This is not a niche concern for one lab. In the twelve months to September 2026 the two largest model providers each published two separate postmortems of exactly this failure. Anthropic's September 2025 postmortem describes three overlapping infrastructure bugs that degraded Claude output quality, peaking at 16% of Sonnet 4 requests on 31 August 2025 [1]. Anthropic's April 2026 postmortem describes three product-layer changes that degraded Claude Code for roughly a month with no model change at all [2]. OpenAI's February 2024 status incident and its April 2025 sycophancy rollback are the same shape from a different company [3][4]. Below the labs, the open-serving ecosystem has spent the year discovering that the same model weights produce materially different quality on different inference engines and hosts.
In every documented incident in this corpus, the detector was a user, never a gate. The dashboards that catch an outage are structurally blind to a quality regression, because quality is not a property of one request and outage monitoring is built per request. Both Anthropic and OpenAI converged on the same remedy, and it is an admission: run evaluations continuously against the true production system, and treat a human tester's "this feels off" as blocking. The pre-ship offline gate, the thing most teams build first, is not where these get caught.
Scope. This guide covers text and code LLM serving where a hosted function is expected to stay fixed and a surrounding change silently degrades it. It covers first-party providers and third-party inference hosts. It does not cover retrieval quality in RAG systems, agent-orchestration bugs above the model, deliberate model version upgrades that change behaviour on purpose, or training-time regressions, each of which is a different failure with a different detection story.
There is no single system that prevents silent regressions. What the published accounts share is a control loop bolted onto the serving path: a way to detect drift, a way to attribute it to a layer, and a way to reverse it. The interesting part is how thin each of those is in practice.
Reconstruct the common shape across the accounts and it separates into two planes. The serving plane is the request path itself, the same five stages as Figure 1, each an independent place where quality can move. The integrity plane is the machinery that is supposed to notice when it does. The lesson of the corpus is that the serving plane is elaborate and the integrity plane is improvised, and that the gap between them is where every incident lived for hours or weeks before anyone acted.
Read the two planes stage by stage, because the design decisions in section 3 are all about which plane you invest in.
Sends a request to a server. The failure is subtle: a request lands on a server configured for a different context length, batch profile or model variant, and comes back degraded rather than failed. Anthropic's largest 2025 incident was exactly this, a short-context request routed to a one-million-token server [1].
Runs this way at: Anthropic, and implicitly at every multi-variant host.
The kernels that do the arithmetic and the sampler that picks the token. This is the densest cluster of silent-failure sources: a fast approximate top-k that is wrong for some batch sizes, a mixed-precision mismatch, a dropped sampling parameter. Correct on average, wrong on a slice.
Runs this way at: Anthropic (TPU), vLLM, SGLang.
The system prompt, context assembly and reasoning-effort settings wrapped around the model. Changing any of them changes the answers. Anthropic's April 2026 Claude Code regression came entirely from this layer, with the model untouched [2].
Runs this way at: Anthropic, GitHub Copilot, every application built on someone else's model.
The test suite run before a change ships. Present at every serious operator, and by itself insufficient: OpenAI's sycophancy update passed offline evals and A/B tests and was still wrong [5]. It catches what it was written to catch and is blind to the property nobody thought to measure.
Runs this way at: GitHub Copilot (4,000+ tests), OpenAI, Anthropic.
The same evals run on the real serving endpoint on a schedule, not on a staging copy. This is the box the postmortems say was missing. Anthropic's remediation commits to running "quality evaluations continuously on production systems" [1]; before the incident, evals ran on a system that did not have the bug.
Named as newly-added at: Anthropic. Rare elsewhere.
The path from a user saying "it got worse" to an engineer reproducing it. At the labs this path is throttled by privacy controls that stop engineers reading the transcripts, which is correct for privacy and catastrophic for detection latency [6].
Runs this way at: Anthropic, and any provider with real privacy controls.
Every box above is placed by a named source. The two planes are my framing, not a term any provider uses; I am calling them the serving plane and the integrity plane because the sources describe the same split in three different vocabularies. Where a box is marked weak or missing, that is the provider's own postmortem saying so, not my inference.
Five forks recur across the accounts. Each has a defensible answer on both sides, and each has a condition that flips it. The conditions are the transferable part.
The other three forks are best read as a table. Each row is a real decision a team recorded, not a hypothetical.
| Decision | Chosen | Rejected | Because / flips when | Evidence |
|---|---|---|---|---|
| Batch-invariant kernels for determinism? | Yes, when reproducibility is required (RL training, evals, audit) | No, keep batch-dependent kernels for peak throughput | Determinism costs 25–45% throughput today. Flips toward invariant kernels when you need bit-identical outputs or defensible evals; stays with fast kernels for latency-bound serving where small output drift is acceptable. | Thinking Machines, SGLang |
| Where does determinism live: the engine's IR, or outside it? | Outside: keep the IR "unaware of batch invariance as a concept", route via platform kernel lists | Make batch-invariant dispatch a first-class IR concern | vLLM's own RFC argues determinism is a platform-and-kernel property, not an architectural one, and closed the first-class proposal as not planned. Flips if a hardware generation makes invariance impossible to bolt on after the fact. | vLLM RFC #40628 |
| Consuming a third-party host: pin the provider and quantization, or take the cheapest route? | Pin, when you are measuring or when quality matters | Auto-route to cheapest, accept whatever precision arrives | Backend and quantization move scores by up to ~16 points on identical weights. A community review found 31 of 32 research codebases used unpinned routing unsafely. Flips toward cheapest only for throwaway or cost-bound workloads where a quality swing is tolerable. | Silent Hyperparameter, LessWrong review, OpenRouter docs |
One thread ties these together. Every "before" column is a speed or cost optimisation that was correct in aggregate and wrong on a slice, and every "after" column trades some of that speed back for a property the team could not otherwise observe. That is the real decision underneath all five: how much performance will you spend to make quality observable? The teams that answered "none" are the ones who wrote the postmortems.
Five failure classes account for every incident in the corpus. Grouping by class rather than by company is the point: the same mistake recurs across organisations that have never seen each other's code.
The classes are: precision and kernel miscompilation, where a fast arithmetic path is wrong for some configurations; silent routing and configuration, where a request is served by the wrong thing and degrades rather than fails; behavioural regression through green gates, where the evals pass because they do not measure the property that broke; product and harness drift, where the model is untouched and the wrapper degrades it; and cross-host serving divergence, where identical weights give different quality on different engines. Each card below is one published incident.
Everything quantitative in the corpus, with its context and date. Read the note below the table before you carry any of these into a design review.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Peak degraded traffic, routing bug | 16% | Anthropic | Sonnet 4 requests, worst hour | Aug 2025 | [1] |
| Claude Code users touched | ~30% | Anthropic | at least one affected message | Aug 2025 | [1] |
| Same bug on Bedrock / Vertex | 0.18% / <0.0004% | Anthropic | platform-dependent blast radius | Aug 2025 | [1] |
| Rollback time, sycophancy | ~4 days | OpenAI | ship (25 Apr) to full rollback (29 Apr) | Apr 2025 | [4] |
| Harness regression duration | ~1 month | Anthropic | Claude Code, product-layer changes | Mar–Apr 2026 | [2] |
| GPT-4 prime-ID accuracy drift | 97.6% → 2.4% | Stanford / Berkeley | same product, Mar vs Jun 2023 | 2023 | [12] |
| Max benchmark swing from backend | 16.6 pp | research survey | identical weights, different engine | 2026 | [13] |
| gsm8k drop from one dropped param | 51.56 → 34.38 | EleutherAI harness | repetition_penalty not applied in vLLM path | Jul 2025 | [16] |
| Tool-call schema accuracy spread | 100% → 83% | Moonshot | same model, across API hosts | 2025–26 | [15] |
| Determinism throughput cost | 25–45% (avg 34%) | SGLang | batch-invariant kernels, vs 61.5% earlier | Sep 2025 | [10] |
| Research codebases routing unsafely | 31 / 32 | community review | unpinned OpenRouter provider | Jul 2026 | [8] |
| Provider-accuracy sampling depth | 32 / 16 / 8 runs | Artificial Analysis | AIME25 / GPQA / IFBench per provider | Aug 2025 | [17] |
Measured, from a primary account: the Anthropic blast-radius figures, the OpenAI and Anthropic timelines, the lm-eval score deltas, the SGLang overhead, and the Chen-Zaharia-Zou drift are all reported by the party that measured them. Independent measurement: the 16.6-point backend swing and the 31-of-32 routing figure come from third-party studies, not from the vendors they implicate. What nobody has published: no provider has disclosed the design or the false-positive rate of a production quality-canary system, how long a silent regression typically runs before a user reports it, or the cost of running continuous production evals at scale. Those are the numbers you will have to generate yourself, which is the subject of section 7.
The primary sources behind this page, graded and filterable. The full ledger,
with one row per claim and the supporting quote copied verbatim, ships beside this file as
sources.md.
Three overlapping infrastructure bugs degraded Claude output quality. The canonical account of the whole failure class: routing, output corruption, and an approximate-top-k miscompilation, none of them a model change.
Three product-layer changes degraded Claude Code for about a month with the model and inference layer confirmed unaffected. The purest example of harness drift.
A user-experience optimisation introduced a bug where "inference kernels produced incorrect results when used in certain GPU configurations", producing nonsense output.
Offline evals and A/B tests were positive; expert testers said the model "felt slightly off" and were overruled. "Unfortunately, this was the wrong call." The definitive green-gate-regression account.
A community mega-report alleging deliberate downgrading. Wrong theory, real signal: shows what detection-by-user-report looks like, and Anthropic's reply that privacy controls "prevent engineers from examining the problematic interactions".
One silently unapplied sampling parameter moved gsm8k from 51.56 (HF) to 34.38 (vLLM). The smallest possible cause with a 17-point effect.
"torch.compile alone is sufficient to break batch invariance on SM89." Determinism is hardware- and compiler-conditional, and the same property broke again on Blackwell in #32992.
The model vendor ships a 4,000-request conformance suite to police its own API hosts, measuring schema accuracy and similarity to the official endpoint. Schema accuracy ranged from 100% to 83% across hosts.
A recorded argument about who owns determinism. The proposal to make it a first-class IR concern was closed as not planned in favour of keeping the IR "unaware of batch invariance as a concept".
RL training is determinism's second customer, with a bitwise requirement: "same logical inputs -> same tokens/images, independent of batch composition and scheduling races".
Corrects the folk explanation for irreproducible output: the cause is batch-size- dependent kernels, and serving load changes batch composition, so identical requests diverge. Ships batch-invariant kernels.
Puts a price on determinism in a production engine: 25–45% slowdown, average 34.35%, down from 61.5% for the first integration, most of it in unoptimised batch-invariant matmul and attention.
A correct engine and a correct model still shipped a broken combination: 218 successful parses of 1,200+ tool calls, under 20%, from three chat-template mismatches. Conformance testing found it; fixes gave a 4x improvement.
Reports Artificial Analysis's provider benchmarks for gpt-oss-120b: after fixes, Groq and Azure reached 93.3% on AIME25 while Vertex sat at 83.3% on identical weights.
A downstream consumer treats offline evaluation as an integration test, over 4,000 tests, re-run "whenever we make a meaningful change" and compared to a baseline. The good-practice version of the gate.
Reviewed influential AI-safety codebases: 31 of 32 used OpenRouter unsafely. After a re-run changing only the provider, one paper's author conceded results "were contaminated by bad inference setups".
The first systematic measurement that a hosted model drifts under the same name: GPT-4 prime-identification fell from 97.6% to 2.4% between March and June 2023.
Backend choice alone shifts benchmark scores by up to 16.6 points; 200 engines surveyed; the inference stack is rarely reported across 35,000 publications.
Detecting provider-side model changes from the outside is tractable: compare output distributions over time. The consumer's defensive counterpart to a provider's canary.
Quantization schemes with similar perplexity "can diverge meaningfully on instruction-following and reasoning benchmarks, so intrinsic metrics alone are insufficient". Perplexity under-detects quality loss.
Lays out the practitioner lifecycle: offline regression testing to gate changes, online monitoring to watch production behaviour between changes. The two-plane split from a vendor-neutral voice.
Continuous third-party measurement of hosted-model accuracy done seriously: GPQA 16x, AIME25 32x, IFBench 8x per provider, reporting median with percentiles across 18 hosts.
The model vendor's own verification recipe for third-party hosts: a smoke test expecting "0 invalid requests and over 90% on both pass@k and pass^k", then AIME25 / GPQA / HealthBench against published scores.
You cannot buy your way out of this class; the postmortems show the largest providers had to build the integrity plane themselves after being burned. The rungs below cross from a weekend script to a production quality canary. The line from toy to real is between rungs 3 and 4.
Send one identical prompt to your endpoint 50 times under varying concurrency and diff the outputs. Then vary batch size deliberately and watch the top token change.
Done when: you can make the same prompt return two different answers by changing only the load. Teaches: why "same request, same answer" is false by default, per Thinking Machines.
Assemble 100–300 prompts with known-good answers spanning the capabilities your users depend on. Write a scorer, exact-match where you can, an LLM-judge where you must, and record a baseline score.
Done when: one command prints a single quality number for the current endpoint. Teaches: quality has to be reduced to a comparable scalar before drift is visible.
Run the golden set in CI on any change to prompt, model, engine, quantization or routing config, and fail the build on a regression past a threshold. This is the GitHub Copilot discipline: re-run on every meaningful change against a fixed baseline.
Done when: a config PR that drops a sampling parameter goes red before it merges. Teaches: the gate is necessary. The next rung is why it is not sufficient.
Pick the failure your gate cannot see, sycophancy, verbosity, format drift, and build a targeted eval for it. This is the direct lesson of the OpenAI rollback: the gate only catches what it measures, so widen what it measures toward what your testers feel.
Done when: the eval flags a behavioural change your generic accuracy score misses. Teaches: every incident adds one metric; the backlog of unmetriced failures is your real risk surface.
Schedule the golden set against the real production endpoint, not a staging copy, every few minutes, and alert on a drop. This is the box Anthropic's remediation added: the pre-ship gate ran on a system that did not have the bug.
Done when: an alert fires from production drift with no deploy on your side, e.g. an upstream provider change. Teaches: detection latency, not gate coverage, is what turns a bug into a month-long incident.
When the canary fires, you need to localise fast. Instrument the request path so you can diff quality by server config, batch size, engine version and prompt version, and bisect the stack rather than the weights, following Figure 3.
Done when: you can attribute a regression to a layer in minutes without reading user transcripts. Teaches: the privacy controls that block transcript access make layer-level telemetry the only fast path.
If you consume third-party hosts, adopt the vendor's conformance suite where one exists (OpenAI's cookbook, Moonshot's K2 verifier) and pin provider and quantization. If none exists, build the black-box change detector from Sam et al. and run it against each host.
Done when: switching hosts or a silent host-side change fails a check before it reaches users. Teaches: for consumed models, the supplier is an uncontrolled part of your stack; treat their endpoint as an untrusted dependency.
The queries that surfaced the material above. The page goes stale in a year; the method does not. Copy and adapt.
"postmortem" degraded model quality inference bug -tutorialsite:status.openai.com OR site:status.anthropic.com "output quality" incident"we never" "degrade" model quality demand load"approximate top-k" OR "top-p" bf16 fp32 wrong token samplingbatch invariance torch.compile "breaks" site:github.com"inference kernels" incorrect results GPU configurationsame model "different results" vllm huggingface backend site:github.cominference provider "inconsistent performance" quantization benchmark"vendor verifier" OR "conformance" tool call schema accuracy providers"how we evaluate" LLM regression CI gate golden set engineering blogcontinuous eval "production" model quality canary drift alertdetect "black-box" LLM API changed over time arxivBuild-environment note. This guide was compiled
in a sandbox whose network egress allows direct fetching only of github.com.
GitHub-hosted sources were fetched in full; every other source was verified through live
web-search result content in this session, with exact quotes and figures copied verbatim,
not reconstructed from memory. The link checker will report the non-GitHub hosts as
unreachable from this environment, which reflects the sandbox, not the links. The two
talk-tier citations carry no timestamp because video and media pages cannot be fetched here;
their claims are taken from the publisher's own page text.