When the model quietly gets worse

A field guide to the production incident where answer quality drops while availability, latency and error dashboards stay green, because the change was in a router, a kernel, a precision mode, a prompt or a third-party host, not in the model. Built from the published postmortems, issue threads and decision records of Anthropic, OpenAI, vLLM, SGLang and others, it gives an architect a reference for the detection-and-attribution control loop, a decision table for the speed-versus-observability trade-offs, a failure catalogue organised by class, and a build ladder from a weekend detector to a production quality canary.

What you get out of it

• The failure is almost always a speed or cost optimisation that is correct on average and wrong on a slice: approximate top-k, an inference kernel, torch.compile, a dropped sampling parameter.
• Request-level dashboards (availability, latency, errors) cannot see quality drift; the only early exit from the latent state is a continuous eval run against the real production endpoint, not a staging copy.
• The privacy controls that protect users also throttle the fastest attribution path, so layer-level telemetry, not transcript access, is what localises a regression.
• For consumed models, the inference engine and quantization are an uncontrolled experimental variable: backend choice alone can move a benchmark by up to 16.6 points on identical weights.

Scope

Why this, now. In the twelve months to September 2026 the two largest model providers each published two separate postmortems of this exact failure, and the open-serving ecosystem spent the year discovering the same model weights give different quality on different engines and hosts.

What it does not cover. Retrieval quality in RAG systems, agent-orchestration bugs above the model, deliberate model version upgrades that change behaviour on purpose, and training-time regressions.