Inference & Serving
23 min
Speculative Decoding: How a Small Draft Model Makes Large Language Models Think Faster
Autoregressive decoding forces even the largest language models to emit one token at a time, bottlenecked by memory bandwidth rather than compute. Speculative decoding breaks this sequential constraint by letting a lightweight dr…