Correctness, within tolerance

There is a file in the vLLM repository called test_speculators_correctness.py. It is the test that stands behind the most consequential promise in modern inference serving: that speculative decoding makes a model faster without making it different. Open it and read what it asserts.

Three things, for each of four draft configurations. That GSM8K accuracy is at least the expected figure less a relative tolerance of three to five per cent. That mean acceptance length is at least its expected value less ten to twenty per cent. That each per-position acceptance rate clears its own floor. Every assertion is one-sided — a greater-than-or-equal against a threshold. The docstring says what it is for: to evaluate "GSM8k accuracy to ensure the speculators-format model produces correct outputs, and check that acceptance length does not collapse under batched inference".

Nowhere in the file does the target model run on its own and get compared with itself.

That is not a bug in the test. It is a description of where the industry has arrived. Speculative decoding is the only optimisation in the inference stack that was adopted on the strength of a mathematical guarantee rather than a measurement, and the guarantee is what allowed a second model to be introduced into the output path of production systems without anyone treating it as a second model. The arithmetic is elegant: a small drafter proposes k tokens, the large target verifies them in one forward pass, and a rejection sampler accepts a prefix in a way that provably reproduces the target's own distribution. You get the same text, sooner. Because the equals sign is proved, nobody has to test it.

The proof is fine. What has quietly stopped being true is that anything in a running deployment can tell you whether the proof still applies to it.

vLLM is admirably candid about the boundary. Its documentation breaks losslessness into three parts: theoretically lossless "up to the precision limits of hardware numerics"; algorithmically validated by a rejection-sampler convergence test and a greedy-equality test; and, third, no guarantee of stable log probabilities at all. The two tests cited for the second claim are linked at pinned commits from an older tree. Neither path — tests/spec_decode/e2e/conftest.py, tests/samplers/test_rejection_sampler.py — exists on main today. The claim is intact; the citations point backwards. What lives at the current path is the acceptance-floor file above.

So the guarantee is asserted in prose and instrumented as a rate. And the rate is a throughput number.

Consider what that does to the three ways the property actually broke this fortnight, in three unrelated projects.

On 18 September an SGLang user reported that the DFLASH speculative path assumes a draft checkpoint's block layout rather than reading it. Serve an anchor-first checkpoint and the server produces draft tokens shifted by one position, with no error and no assertion. Acceptance collapses to approximately 1.0 — only the anchor survives. The reporter measured the official drafter against a Qwen3.8-Flash-Next target at 1.0621 accepted tokens per four-token window on SGLang, against 2.7562 to 2.9837 for the same pair under vLLM. The checkpoint declares its convention in a field called query_zero_predicts_next. The code does not read it. The server, in the reporter's words, "reports healthy".

That failure did announce itself — in the acceptance number, which is the only place it could. And an acceptance number two-thirds below expectation reads, to everyone who looks at it, as a tuning problem. It goes in the performance queue. The correct reading was that a dependency's interface had never been checked at load.

On 5 September a vLLM contributor opened a fix for a worse case: pipeline parallelism at PP≥2 with multi-token prediction and prefix caching on hybrid models, with more than two concurrent sequences. Stale batch-row indexing let one request's block table address another's block IDs and poison recurrent state through aliased cache regions. The symptom: 14 to 33 per cent of requests degenerating into a constant-token loop, emitting token 1023 sampled from all-NaN logits, with acceptance falling from roughly 50 per cent to 26. On a four-way pipeline the author measured 16 of 48 requests looping before the fix and 0 of 56 after, mean acceptance length rising from 2.80 to about 4.95. Note that vLLM's own documentation lists pipeline parallelism as not composable with speculative decoding in versions up to 0.15.0. Somebody was running it in production anyway, which is how the bug got found.

And on 13 July — still open — a llama.cpp user filed the one that has no instrument at all. With temperature 0 and top_k 1, draft-model speculation on a Q4_K_M target produced different greedy output from the same target run without speculation. The same draft method on the bf16 target matched on all four prompts. An n-gram method on the quantised target stayed lossless. Two families of drafter, two model families, one pattern: the interaction sits between draft-model speculative paths and quantised target computation. The invariant the reporter wrote down is the whole contract in one line — greedy speculative output equals greedy vanilla output for the same target weights, seed and prompt — and the only way it surfaced was a human running both sides by hand and diffing them.

Nothing in any of these stacks runs that comparison. There is no metric for it, because a metric for it would require computing the answer twice, which is exactly the cost speculation exists to avoid.

Look at what is exported instead. llama.cpp's server publishes draft tokens, accepted tokens, verification steps, and accepted tokens per draft position. vLLM's per-request acceptance metrics — experimental, and off by default — give mean acceptance length, draft acceptance rate, an acceptance histogram and step counts. Every field is a quantity of acceptance. Not one of them distinguishes a request whose output matched the target from one whose output did not, and there is no field that could, because acceptance is a count of what the verifier admitted, and every one of these faults corrupted the verifier's inputs rather than its logic.

The strongest objection is that this is a bug story dressed up as an architecture story. Three defects, three fixes, reported by users and triaged by maintainers: this is what a stack looks like while it matures, and the rejection-sampling theorem is not weakened by an indexing error in a Mamba block table. Nobody ever promised bit-identical text from an LLM endpoint anyway. llama.cpp's own server documentation warns that prompt cache reuse — default on — can cause nondeterministic results because logits are not guaranteed bit-for-bit identical across batch sizes. I argued ten days ago that batch composition therefore belongs to a serving system's output contract rather than its transport; I am returning to it here only because speculative decoding is the place where that concession gets collected. If output already varies, an architect demanding equality is demanding something they never had.

That objection is right about the theorem and wrong about the consequence. Once variation is normalised, there is no threshold above which a real defect declares itself. A fault that changes 33 per cent of your completions is indistinguishable, from the outside, from the tolerance you already agreed to live with — and the number you would notice it in is the number your platform team is paid to raise. That is the specific trap. Everywhere else in engineering, a correctness control and a performance control point in opposite directions, and the tension is visible in who signs off. Here they are the same reading, and the incentive on it runs one way.

It gets more explicit than that. vLLM's adaptive verification sizes how much of each draft to verify from the drafter's own per-position confidence, admitting slots against a budget chosen by a cost model profiled at startup to maximise expected accepted tokens per second. That is a legitimate feature and not a correctness compromise — verifying less of a draft is equivalent to speculating less, and the target still produces every token it emits. What it does mean is that verification depth is now an economic variable with an objective function, and the objective function is throughput. Meanwhile both projects ship a way to manufacture the gauge reading directly: llama.cpp's --spec-synth-len and --spec-synth-rates are labelled "benchmarking only", and vLLM's equivalent rejection_sample_method: synthetic, with per-position unconditional acceptance rates, sits in the same user-facing configuration table as the key that turns speculation on, carrying no such label.

The practical response is not to switch speculation off. It is to run, in your own environment, the test the projects cannot run for you: same prompts, greedy, speculation on and off, on your quantised target, your parallelism topology, your batch size, as a release gate rather than a curiosity — and to put the draft checkpoint under the same version pinning and evaluation gate as the target, because it is a model in the serving path that happens to have arrived through a performance ticket. Then alarm on acceptance falling, and route that alert to whoever owns output quality rather than whoever owns latency.

Everything above is argued from the projects' own public record — their documentation, tests, issues and patches — because that is the evidence that can be read directly rather than through someone's summary of it. It is also, conveniently, the evidence a reviewer can check.

GSM8K accuracy within three per cent is a perfectly reasonable test of a model. It is a strange thing to have become the test of an equals sign.