Speculative Decoding

Decode is sequential: each output token needs a forward pass of the whole model, and each pass reads all the weights from memory to produce one token. A single-stream request therefore uses a small fraction of the accelerator's arithmetic capacity while saturating its memory bandwidth.

Speculative decoding spends that idle compute. A small draft model proposes k tokens; the target model then verifies all k in one forward pass. Accepted tokens are effectively free, and the first rejected token is replaced by the target's own sample while the remainder of the draft is discarded.

With the correct verification procedure the output distribution is identical to sampling from the target model alone. That is the property that makes this an optimisation rather than a quality trade, and it is what distinguishes it from simply using the small model. The technique was introduced in published work in 2023 and is now standard in major serving stacks.

Why it matters

It is one of the few techniques that improves interactive latency without changing the model the user gets. Reported gains are roughly 2 to 3 times tokens per second for single-stream or low-concurrency serving, which for a coding assistant or a chat product is the difference users actually notice.

It also has a sharp and frequently missed boundary: it is a latency technique, not a throughput technique. Teams enable it fleet-wide, measure aggregate tokens per second, see no improvement or a regression, and conclude the technique does not work - when the real finding is that their batch sizes had already removed the idle compute it exploits.

Implementation patterns

• Pick the draft model for acceptance rate, not for size. A model distilled from or trained alongside the target accepts well; a generic small model on specialised text - code, legal, another language - may accept so little that the technique is a net loss.
• Measure acceptance on your own traffic before committing. 80% at k = 4 and 30% at k = 4 are different products.
• Tune k against acceptance. High acceptance justifies longer drafts; low acceptance means a long draft mostly wastes compute. Typical values are 3 to 8.
• Enable it per route, not per deployment, so the interactive path has it and the batch path does not.
• Monitor acceptance rate and tokens per second per route as live metrics. Acceptance drifts when the traffic mix changes, and nothing else will tell you.
• Account for the draft model's memory. It occupies accelerator memory that would otherwise hold key-value cache, and cache capacity is what sets maximum batch size.
• Consider n-gram or prompt-lookup drafting where output repeats input heavily, such as code editing or summarisation with quotation: it needs no second model at all and accepts well on that traffic.

Industry example

Every major open serving stack has shipped an implementation since 2023 - as speculative decoding, assisted generation or an equivalent name - and the published guidance converges on the same caveat: gains are largest at low batch sizes and shrink as concurrency rises. Hardware vendors' inference libraries document the same boundary, because the mechanism depends on an arithmetic-to-bandwidth imbalance that large batches remove.

Failure scenarios

• Enabled under heavy batching, where rejected drafts consume capacity that would have served other users' tokens, so aggregate throughput falls.
• Draft-target mismatch, giving a 25% acceptance rate and a net slowdown with more hardware consumed.
• Draft memory squeezing the key-value cache, lowering maximum batch size and therefore throughput - a second-order effect that is easy to miss when only latency is measured.
• Verification implemented incorrectly, which silently changes the output distribution. The symptom is a quality regression with no quality-related change in the change log, and it is extremely hard to attribute.
• Acceptance drift. A product expands into a new language or domain, acceptance falls, and nobody is watching the metric that would show it.
• Treated as a cost optimisation. It raises total compute per token; it reduces wall-clock time. Presenting it as a saving sets up a disappointing invoice.

Trade-offs

When not to use it

On a throughput-maximising batch service, where concurrency already fills the arithmetic units. Measure before assuming; the crossover depends on model size, hardware and sequence lengths.

On a hosted API, where you do not control the serving stack. The provider may already use it; either way it is not a lever available to you, and the equivalent levers are prompt structure and output length.

Where accelerator memory is the binding constraint and batch size is already limited by key-value cache, since the draft model competes for exactly that memory.

And where output is extremely short - a classification or routing call producing five tokens - because the fixed overheads dominate and there is almost no sequential chain to shorten.

Interview question

Q: A team reports that enabling speculative decoding made their service slower in aggregate, and proposes removing it. What do you ask, and what do you expect the answer to be?

What a strong answer covers: the mechanism - draft proposes, target verifies k tokens in one pass, so it converts sequential steps into parallel compute · asking for batch size and concurrency first, because the gain exists only while compute is idle · asking for the acceptance rate, and whether the draft is distilled from the target · the draft model's memory competing with the key-value cache and so with batch size · the recommendation to keep it per route, on interactive traffic, off the batch path · that it costs more compute per token and buys wall-clock latency, so it is not a cost optimisation · and the correctness point that verification preserves the target's distribution only if implemented properly, which is worth checking if quality has also moved.

Quick check

Quiz: Why does speculative decoding help a single user and not a saturated batch service? Because it spends arithmetic capacity that decode leaves idle at low concurrency, and large batches already use that capacity - so speculation then competes for the same compute instead of exploiting a gap.

Flashcard: What does speculative decoding cost, and what does it not cost? - It costs extra compute per token, draft-model memory taken from the key-value cache, and a second model to operate; it does not cost output quality, because verification preserves the target model's distribution exactly.