Silent Quality Regression

Support tickets say the product got worse. Every dashboard is green: no error-rate change, no latency change, no throughput change, no deploy correlated with the complaints. The defect is inside the payload, and nothing in the observability stack asserts on the payload.

This class of failure is not confined to machine-learning systems, though that is where it is most acute. A ranking change that quietly worsens relevance, an OCR pipeline whose accuracy drops after a library upgrade, a pricing service returning plausible but wrong numbers, a recommendation model trained on a broken feature - all return 200 within budget.

Why it matters

Time-to-detection is measured in weeks rather than minutes, because the loop that normally closes it is absent. For a crashing service, the system tells you. Here, users tell you, slowly, in language ("it feels worse") that is hard to distinguish from noise and easy to dismiss.

The compounding factor is dilution. A defect affecting a minority of traffic is invisible in any average. A bug touching 16% of one model's requests at its peak does not move an aggregate quality number, and it does not appear at all in a per-endpoint error rate. You find a subpopulation only if you are already slicing by the dimension that separates it - and that dimension is usually infrastructural (which accelerator, which region, which routing decision), while dashboards are usually built along product dimensions.

Implementation patterns

• A continuously scored golden set. A fixed, held-out population of representative requests, replayed against production on a schedule and scored automatically. This buys the correctness oracle you cannot have per request, at the price of sampling.
• Slice quality by infrastructure dimensions, not only product ones: hardware type, serving version, region, routing path, compiler or runtime version. The defect lives where nobody is looking.
• A pre-built reproduction path that respects your privacy controls - consented debug traffic, synthetic replays that mirror real request shapes, or an audited break-glass procedure. Designing this during an incident costs days.
• Treat qualitative reports as a leading indicator with a threshold, routed and counted rather than answered individually.
• Version and re-baseline deliberately. A quality time series is meaningless without a stable reference, and the reference must survive model, prompt and traffic-mix changes.

Industry example

Anthropic's published postmortem covering August to early September 2025 describes three overlapping infrastructure bugs that degraded Claude's output quality: a context-window routing error that at its worst hour on 31 August affected 16% of Sonnet 4 requests, an output corruption traced to a misconfiguration on the API's TPU servers, and a latent miscompilation in the approximate top-k path of the XLA:TPU compiler that affected Haiku 3.5 for nearly two weeks. None produced an error.

The postmortem also names a detection obstacle worth studying: internal privacy and security controls limited engineers' access to the user interactions needed to reproduce the reports. The control was correct on its own terms and removed the fastest diagnostic path, which is the shape of a genuine trade-off rather than a mistake.

Failure scenarios

• The regression rides in on a dependency. A compiler, kernel, driver or library upgrade changes numerical behaviour; nothing in the application changed, so the deploy correlation that usually localises a bug is absent.
• Partial traffic exposure. A subset of servers with a different configuration serves worse results, and aggregate metrics average it away.
• Baseline drift. The quality series is compared against a moving reference, so a slow decline reads as normal.
• The metric measures the wrong thing. Automated scores that correlate with quality in the common case stop correlating exactly where the defect lives.
• Reports dismissed as subjective. The single most common organisational failure: the earliest and cheapest signal is discarded because it lacks a number.

Trade-offs

Golden-set scoring costs money continuously - inference or compute on traffic that serves no user, plus the engineering to keep the set representative as the product changes. It also creates a metric people will optimise, with the attendant risk of improving the score rather than the product.

The counterweight: the alternative is learning about quality regressions from customers, on their timeline, which in a competitive market is paid in churn rather than in compute.

When not to use it

When your output has a cheap, genuine correctness oracle. A service that can assert on a schema, a checksum, a reconciliation, or a total that must balance should build that assertion and skip the statistical layer entirely - it is stronger, faster and cheaper.

Also when the output is not graded but binary, and when volume is low enough that every response is already reviewed by a human. The machinery earns its cost specifically where correctness is a matter of degree: ranking, recommendation, generated text, transcription, extraction, forecasting, risk scoring.

Interview question

Q: Users report that your search results have got worse over the past three weeks. Error rate, latency and click-through are all within their normal ranges, and no ranking change shipped in that window. How do you investigate, and what would you have built beforehand to make this a ten-minute question?

What a strong answer covers: recognising that envelope metrics cannot see payload defects, so the investigation must start from a scored population rather than from dashboards · slicing by infrastructure dimensions including index build version, serving version and hardware · that click-through is itself diluted and lags · the golden set replayed on a schedule as the thing that should have existed · the baseline problem, and why a moving reference hides slow declines · and the honest limit: a statistical detector needs enough volume per slice to be significant, which bounds how fine the slicing can go.

Quick check

Quiz: Why does a defect affecting 16% of one model's requests stay invisible on a quality dashboard? Because it is diluted in any aggregate, and it is only separable along an infrastructure dimension - routing, hardware, serving version - that product dashboards do not slice by.

Flashcard: What is the expensive part of quality monitoring - the detector or the baseline? — The baseline. Scoring is cheap; keeping a stable, representative reference that survives model, prompt and traffic-mix changes is the standing cost.