Inference Optimisation
KV cache, FlashAttention, speculative decoding, quantisation and continuous batching.
7concepts
33flashcards
60minutes of reading
- 01 FlashAttention An IO-aware attention kernel that is both faster and lower-memory than the textbook implementation by tiling computation to keep activations in SRAM.
- 02 Mixture-of-Experts Inference Why serving MoE models is harder than serving dense models of equivalent quality, and how DeepSeek and Mistral made it work in production.
- 03 Paged Attention and the KV Memory Manager How vLLM's PagedAttention treats the KV cache like OS virtual memory, cutting fragmentation waste from 60-80% to under 4% and roughly doubling to quadrupling serving throughput.
- 04 Speculative Decoding Use a small draft model to propose tokens that a large verifier accepts or rejects in parallel, giving lossless 2-3x latency wins on autoregressive generation.