Inference Optimisation
KV cache, FlashAttention, speculative decoding, quantisation and continuous batching.
7concepts
33flashcards
60minutes of reading
- 01 KV Cache Why decoder inference is quadratic without a KV cache and linear with one, and why managing that cache is now the dominant memory problem in LLM serving.
- 02 Quantisation - INT8, INT4, FP8 How to cut weight and activation precision below 16 bits without wrecking quality, and which scheme to pick for which deployment.
- 03 vLLM and Continuous Batching Why static batching wastes most of your GPU on variable-length workloads, and how iteration-level scheduling combined with PagedAttention raises throughput by an order of magnitude.
- 04 FlashAttention An IO-aware attention kernel that is both faster and lower-memory than the textbook implementation by tiling computation to keep activations in SRAM.
- 05 Mixture-of-Experts Inference Why serving MoE models is harder than serving dense models of equivalent quality, and how DeepSeek and Mistral made it work in production.
- 06 Paged Attention and the KV Memory Manager How vLLM's PagedAttention treats the KV cache like OS virtual memory, cutting fragmentation waste from 60-80% to under 4% and roughly doubling to quadrupling serving throughput.
- 07 Speculative Decoding Use a small draft model to propose tokens that a large verifier accepts or rejects in parallel, giving lossless 2-3x latency wins on autoregressive generation.