Speed goes to whoever is easiest to predict

A field appeared in vLLM's API responses this month. The example value printed in its own documentation is 0.07751937984496124.

The field is draft_acceptance_rate, and in that example a small model proposed 129 tokens across 43 decoding steps and the large model kept ten of them. The other 119 were computed and thrown away. Before anyone quotes the number: it is an illustration in a documentation page, not a benchmark, and the configuration it illustrates is n-gram matching, the cheapest drafting method vLLM ships. What matters is not the size of the number. It is that there is now a field for it. You do not build per-request reporting for a constant. The feature shipped in v0.29.0 on 9 September behind --per-request-spec-decode-metrics, it is marked experimental, and it reports only for single-sequence requests.

Speculative decoding has been sold for three years as a speedup. The EAGLE repository, whose drafting heads are the ones most serving stacks actually load, advertises 3x faster than vanilla decoding at 13B, and 4x for EAGLE-2. Those numbers were measured and I have no reason to doubt them. But speculative decoding never made a model faster. It converts a serving machine's idle arithmetic into lower latency, and batching spends that same idle arithmetic on other people's requests. The two compete for one pool. Over the past four months the stack has quietly built the machinery to ration what is left: the speedup now goes to the most predictable requests first, and to nobody once the machine is full.

The mechanism is worth getting exactly right, because it explains the rest.

Generating one token from a transformer requires reading every weight out of high-bandwidth memory into the chip. For a model with P parameters stored at two bytes each, a decode step moves about 2P bytes and performs about 2P floating-point operations, one multiply-accumulate per weight. On any current accelerator the bytes take far longer than the operations. The arithmetic units sit idle waiting for memory. This is the ordinary condition of single-stream decoding, and it is why a chip advertised in petaflops delivers you tokens at the pace of its memory bus.

Speculation exploits the idleness. A small drafter proposes K tokens; the large model evaluates all K+1 candidate positions in a single forward pass. That pass reads the weights exactly once, the same read it was going to do anyway, so the extra positions ride along nearly free. If the drafter guessed well you emit several tokens for the price of one trip through memory. vLLM states the condition without decoration: at batch size 1 "the GPU is memory-bound with spare compute, so the extra draft tokens are close to free."

Now notice what else fills spare compute. Continuous batching does, by putting two hundred other users' tokens into the same weight read. Speculation and batching are two answers to the same idleness, and there is only one pool of it. vLLM's documentation, again without decoration: at batch size 256, "draft tokens now compete with real tokens for the same compute, and every rejected token is compute wasted."

So the effective work per step is not the batch size but the batch size multiplied by the draft length. Past a critical point the multiplication stops paying. In June, vLLM merged Dynamic Speculative Decoding, which takes a table mapping concurrency ranges to draft lengths. The example in the documentation reads: K=3 for concurrency 1 to 64, K=1 for 65 to 128, and for 129 to 512, in the docs' own words, "K=0 will be used ... i.e., no draft tokens will be produced." A feature whose headline claim is a 3x speedup, shipped with a configuration schema whose job is to switch it off.

In August the rationing got finer, and more interesting. Adaptive verification, merged on 12 August and extended on the 25^th, scores every individual draft slot rather than every request. Each (request, position) pair gets a survival probability, the running product of that request's per-position confidences, and the highest-scoring slots are admitted until a global budget is spent. The budget comes from a cost model profiled at server startup, which measures what a step costs at each shape and then "picks the token count that maximizes expected accepted tokens per second."

One sentence in that documentation is the whole argument of this piece: "Slots compete across requests: position 5 of a confident request can outrank position 1 of a doubtful one."

That is an auction. The currency is the drafter's confidence in your text, the auctioneer is a cost model, and the objective it maximises is accepted tokens per second across the machine rather than the latency of any request in it. A request the drafter finds easy keeps its full block. A request the drafter finds hard is trimmed after a token or two and decodes at the ordinary, memory-bound rate. Nobody is slowed below baseline. It is the discount that is allocated, not the penalty.

Predictable, here, does not mean dull. The clearest evidence is the third-party leaderboard EAGLE's own README points to as its certification. On a single RTX 3090, Vicuna-7B, greedy, FP16, EAGLE-2 reaches 2.66x on mathematical reasoning and 1.78x on translation. Same model, same card, same method, a factor of one and a half between two tasks. Chains of symbolic reasoning are formulaic in exactly the way a drafter can learn; translation is steered by a source text the drafter has to keep re-reading. NVIDIA's documentation makes the same distinction from the other side when it describes its suffix-automaton drafter as "very accurate when it matches (exact pattern repetition), while neural methods are better for novel content."

That leaderboard is also where the quoted speedups come from, and it carries a settings line that almost nobody repeats: batch size = 1. It was last updated in April 2025. EAGLE's README, meanwhile, mentions batch size nowhere at all.

So a quoted speculative decoding speedup has three hidden arguments: which task, which hardware, and how many other people are on the box. The leaderboard declares two of them, carefully. The third is the one production changes hour by hour.

There is one more thing being traded, and it is worth watching. Speculation earned its place by being free of consequences: sampling from the drafter and verifying against the target is a rejection sampling scheme that reproduces the target's distribution. vLLM describes the property precisely as "theoretically lossless up to the precision limits of hardware numerics," and backs it with rejection-sampler convergence tests and greedy-equality tests. EAGLE makes the same claim for its method. That guarantee is why the feature could ever be switched on by default. It is also, now that the free arithmetic has run out, the next asset on the table. TensorRT-LLM's multi-token-prediction config offers use_relaxed_acceptance_for_thinking, which during a reasoning model's thinking phase will accept a draft token if it merely appears in a top-k candidate set, filtered by a log-probability distance from the top token. That is acceptance of tokens the target model would not have sampled, enabled specifically in the region of the output that is not meant to be read.

The strongest case against all of this is that it is simply good engineering, and I think that case is mostly right. Every optimisation in systems is workload-dependent; nobody signed anything promising a constant. vLLM's own method table says plainly that "real gains depend on your model family, traffic pattern, hardware, and sampling settings." And the alternative is visible in the same fortnight's documentation: TensorRT-LLM notes that there is "currently no way to dynamically disable speculation, thus speed ups are only observable at low batch sizes." Dynamic K and adaptive verification are the fix for that, not the symptom. Low concurrency is precisely when a human being is waiting on a first sentence, so buying latency there and surrendering it under load is the correct trade, made by a cost model optimising the right quantity for a shared machine.

I agree with all of it, and it leaves the problem where it was. The engineering adapted; the noun did not. "A 3x speedup" is a phrase that describes a property of a method. What these three features describe, between them, is a property of a method, a load level, and the text being generated. The field built the adaptive machinery and went on quoting the static number.

Four things follow for anyone learning to serve models. Never accept a speculative decoding speedup without the concurrency it was measured at; where it is unstated, assume one. If you run a server, turn the per-request metrics on and read the histogram rather than the mean: the documentation's own example is [39, 1, 0, 3], which is 39 steps that accepted nothing and three that accepted everything, a bimodal workload that a mean acceptance length of 1.23 conceals completely. Learn arithmetic intensity properly, because batching, speculation, quantisation and sparse-expert serving are four ways of spending one pool of idle compute, which is why their speedups cannot be added together. And stop treating tokens per second as a property of a model.

A caveat on the evidence. Everything above comes from the source repositories themselves, read today: vLLM's feature docs and release notes, NVIDIA's, EAGLE's, and one public leaderboard last updated in April 2025. Four projects, but all of them describing their own work, and none of it independent measurement. I have not run these configurations, and the acceptance figures quoted from documentation are illustrations rather than results.

The drafter is a small model whose entire job is to be unsurprised by a larger one. Its acceptance rate is therefore the cheapest available estimate of how much of your output was predictable by something far smaller than the thing producing it. That quantity has been computed inside every speculative server for years and reported to no one. It now comes back in the response body, marked experimental, to anyone who asks for it. Very few people will ask.