Inference Optimisation
KV cache, FlashAttention, speculative decoding, quantisation and continuous batching.
14concepts
170flashcards
115minutes of reading
- 01 Activation Outliers and Rotation-Based Quantisation A handful of channels in every transformer carry activations a hundred times larger than the rest, which is why activation quantisation fails where weight quantisation succeeds, and why rotating the hidden state fixes it.
- 02 Early Exit and Adaptive Depth Not every token needs all eighty layers, and the methods that act on this observation break batching and the KV cache in ways that decide whether the idea survives contact with a serving stack.
- 03 FlashAttention An IO-aware attention kernel that is both faster and lower-memory than the textbook implementation by tiling computation to keep activations in SRAM.
- 04 KV Cache Eviction and Compression Once the KV cache outgrows HBM the only remaining lever is to keep less of it, and the choice between dropping tokens, quantising them, or skipping them per query decides which capability you lose first.
- 05 Mixture-of-Experts Inference Why serving MoE models is harder than serving dense models of equivalent quality, and how DeepSeek and Mistral made it work in production.
- 06 Paged Attention and the KV Memory Manager How vLLM's PagedAttention treats the KV cache like OS virtual memory, cutting fragmentation waste from 60-80% to under 4% and roughly doubling to quadrupling serving throughput.
- 07 Speculative Decoding Use a small draft model to propose tokens that a large verifier accepts or rejects in parallel, giving lossless 2-3x latency wins on autoregressive generation.
- 08 Ternary and Extreme Low-Bit Models Below about three bits, quantising a trained model stops working and you have to train in low precision from the start, which changes the arithmetic of inference from multiply-accumulate to add-subtract.