A team enables speculative decoding with a small draft model to cut inter-token latency on a chat product. What does it buy, what does it cost, and at what point does it stop helping?

The mechanism, briefly

A small draft model proposes k tokens. The large target model then verifies all k in one forward pass instead of k passes. Accepted tokens are effectively free; the first rejected token is replaced by the target's own sample and the rest of the draft is discarded. With the correct verification procedure the output distribution is identical to sampling from the target alone, which is what makes this an optimisation rather than a quality trade. The technique was introduced in the speculative decoding work published in 2023.

What is gained, quantified

It converts a sequential, memory-bandwidth-bound process into a batched one. During decode, each step reads the whole model's weights from memory to produce one token, so a single-stream request uses a small fraction of the accelerator's compute. Verifying k draft tokens in one pass uses that idle compute.

Reported speedups in the original work and in subsequent serving implementations are roughly 2 to 3 times tokens per second for single-stream or low-concurrency serving, with the gain set by the acceptance rate: at 80% acceptance with k = 4 most of the draft survives, and at 30% most of it is thrown away.

What is paid

• Extra compute per output token, always. Every rejected draft token is work that produced nothing.
• Memory for the draft model on the same accelerator, which is memory not available for the key-value cache - and the cache is what sets the maximum batch size.
• Operational surface: a second model to version, evaluate and deploy, and a tuning parameter (k) whose best value depends on the workload.
• An acceptance rate that depends on domain match. A draft model distilled from the target performs well; a generic small model on a specialised domain - code, legal text, another language - can accept so little that the technique is a net loss. Measure acceptance on your own traffic before committing.

When it stops helping

When the server is already running large batches. Batching large numbers of concurrent requests also fills the idle compute, so decode shifts from memory-bandwidth-bound to compute-bound - and speculation now competes for the same FLOPs it was exploiting. At high concurrency the gain shrinks towards nothing and can invert, because the rejected drafts consume capacity that would have served other users' tokens.

That gives a clean decision rule: speculative decoding is a latency technique for low-concurrency serving, not a throughput technique. Use it where a single user's tokens-per-second is the product (an interactive assistant, a coding tool at low load, an on-device model), and leave it off where the objective is tokens per accelerator per second across many users.

When this is the wrong technique

On a throughput-maximising batch service, where concurrency has already filled the arithmetic units. On a hosted API, where the serving stack is not yours. Where accelerator memory already limits batch size, since the draft model competes for exactly that memory. And where output is five tokens long, because there is almost no sequential chain to shorten. It is also not a cost optimisation - it raises total compute per token and buys wall-clock latency, so presenting it as a saving sets up a disappointing invoice.

How to keep the option to reverse

Make it a per-route configuration flag, not a deployment property, and keep the acceptance rate and tokens-per-second per route as monitored metrics. Then it can be disabled for the batch path and kept on the interactive path - which is usually the right final state - and re-evaluated when traffic patterns change rather than when somebody remembers.