Approximate Operator Hazard
also called Configuration-Dependent Correctness, Approximate Kernel Risk
The risk that an operator chosen for speed - approximate top-k, approximate nearest neighbour, a quantised or fused kernel - produces wrong results only at certain shapes or configurations, so correctness becomes a property of production traffic rather than of the code.
Inference stacks are full of operators whose names announce the trade: approximate top-k, approximate nearest neighbour, fast attention, fused kernels, quantised matrix multiplication. Each exchanges exactness for speed, and each has an error profile that depends on shapes the application does not choose - batch size set by concurrent traffic, sequence length set by user input, the particular model variant being served.
That dependency is the hazard: a unit test at the default shape passes while production, running a distribution of shapes, is sometimes wrong. From the application's point of view the defect is non-deterministic and load-correlated - present at some traffic levels and absent at others, on identical inputs. A defect invisible at batch size 1 and present at batch size 16 is the normal case.
And the failure is quiet. These operators sit on the path that produces a plausible answer, so when they misbehave the result is wrong without being invalid - no exception, no status code, no schema violation. A defect that drops the highest-probability token on 5% of requests is invisible to error-rate monitoring, and a share governed by batch size can sit near zero in a test while reaching tens of percent at peak.
Why it matters
Organisations separate "model quality" from "infrastructure", and this hazard crosses the line: a compiler flag, a kernel choice or a library upgrade changes the product's output. The owners of the evaluation gate are usually not the owners of the serving stack, so the change ships without one.
Detection is structurally hard. A model recovers from isolated bad tokens and a vector index returns some neighbours, so the damage is a small quality decrement spread across many responses - invisible to error-rate monitoring and nearly invisible to noisy quality evaluations, which is why incidents of this class are measured in weeks. And the risk is growing: every generation of serving optimisation adds operators of this kind, because that is where the remaining performance is.
Implementation patterns
- Keep an exactness canary: 30 to 60 fixed prompts or queries with deterministic sampling, compared token for token (or neighbour for neighbour) against a recorded reference, run continuously on every serving configuration. Cheap, and it catches the whole class.
- Assert equivalence across the production shape distribution, not the default shape: the batch sizes, sequence lengths and model variants actually served.
- Treat the serving stack as part of the model's change log: compiler version, kernel selection, operator flags, quantisation settings and library versions, recorded alongside model and prompt versions.
- Gate serving-stack changes on the same evaluation suite as prompt and model changes.
- Run continuous quality evaluation against live traffic, not only pre-deployment, since the signature is a distribution shift rather than a failure.
- Keep the exact implementation available behind a flag, so a suspected operator can be swapped out in minutes to confirm or exclude it.
- Be suspicious of masking workarounds, which leave a latent defect to resurface at a different configuration.
Industry example
Anthropic's postmortem of 17 September 2025 describes a change deployed on 25 August to improve token selection which triggered a latent bug in the XLA:TPU compiler: an approximate top-k operation sometimes dropped the highest-probability token, but only for certain batch sizes and model configurations, and a December 2024 workaround had been masking the bug. It was confirmed on one model, judged likely on two others, and could not be reproduced on one of those. Rollbacks ran from 4 to 12 September and the fix was exact top-k with enhanced precision. The same postmortem lists the detection failures plainly - existing evaluations did not capture the degradation, reliance on noisy evaluations was too heavy, and early reports were hard to separate from normal feedback variation - and the stated remediations were more sensitive evaluations and continuous quality checks on production.
Failure scenarios
- Quality regression with no quality-related change. The change log shows an infrastructure upgrade, so the investigation starts in the wrong place.
- Load-correlated wrongness. Worse during peak hours because batch sizes differ, which looks like a capacity problem.
- Non-reproducible user reports, because the shape that triggers it is not the shape a developer uses, leading to reports being dismissed.
- Post-filtered approximate search, where a selective filter applied after the search returns almost nothing for small partitions - the same hazard in the retrieval stack.
- Quantisation accepted on an average metric, while a specific subpopulation degrades badly.
- A workaround that hides a latent bug for months, removing the history that would have made the later diagnosis quick.
Trade-offs
| Choose | Gains | Pays |
|---|---|---|
| Approximate or fused operators | Substantial speed and cost improvement, which is why they exist | Correctness becomes configuration-dependent; needs an exactness check and a shape sweep |
| Exact operators | Correctness is a property of the code | Slower and more expensive, sometimes by a lot |
| Approximate with an exactness canary | Most of the speed with the class detected | The canary must be built and maintained, and somebody must own it |
When not to use it
This is not an argument against approximation: approximate nearest neighbour is the only way to serve vector search at scale and quantisation is the difference between one accelerator and four. The discipline is the check, not abstinence.
It is the wrong frame where output is cheaply verifiable by construction - a parse, a checksum, a constraint solver's answer - because wrongness is detectable downstream. And on a hosted API the operators are not yours, though the hazard remains, so the transferable control is a fixed reference set scored continuously from outside.
Interview question
Q: Users report that your assistant has been slightly worse for two weeks. Error rates, latency and traffic are all normal; no model or prompt change shipped. How do you investigate?
What a strong answer covers: scoring a fixed reference set against recorded outputs to establish whether quality actually moved, before investigating causes · widening the change log to the serving stack - library, compiler, kernel, quantisation and configuration changes - and treating them as candidate causes · looking for correlation with batch size, sequence length, model variant or time of day, since configuration-dependent defects are load-correlated · swapping the exact implementation in behind a flag to confirm or exclude an operator · why error-rate monitoring could never have caught it, because the output is wrong without being invalid · and the lasting fixes: an exactness canary across production shapes, continuous quality evaluation on live traffic, and serving-stack changes gated by the same evaluation suite as prompt changes.
Quick check
Quiz: Why can a serving-stack optimisation produce a quality regression that unit tests and error-rate alerts both miss? Because an approximate operator's error depends on shapes such as batch size that traffic determines rather than the test, and its wrong output is fluent and schema-valid, so nothing raises an error.
Flashcard: What is the cheapest control for configuration-dependent correctness defects? - An exactness canary: a small fixed input set with deterministic sampling, compared token for token against a recorded reference, run on every serving configuration continuously.