Inference Optimisation
KV cache, FlashAttention, speculative decoding, quantisation and continuous batching.
14concepts
170flashcards
115minutes of reading
- 01 Evaluating Quantised Models A quantised model is meant to be a drop-in replacement, so matching aggregate accuracy is the wrong acceptance test; what you need to measure is how often its answers differ from the original's.
- 02 KV Cache Why decoder inference is quadratic without a KV cache and linear with one, and why managing that cache is now the dominant memory problem in LLM serving.
- 03 Quantisation - INT8, INT4, FP8 How to cut weight and activation precision below 16 bits without wrecking quality, and which scheme to pick for which deployment.
- 04 Structured Pruning and Sparsity Why removing half a model's weights is easy and making it run twice as fast is not, and how the field converged on layer, head, and channel removal followed by distillation.
- 05 Weight-Only Post-Training Quantisation Compressing weights to four bits while leaving activations in bf16 is the default way large models are made to fit, and GPTQ and AWQ get there by two genuinely different arguments about which errors matter.
- 06 vLLM and Continuous Batching Why static batching wastes most of your GPU on variable-length workloads, and how iteration-level scheduling combined with PagedAttention raises throughput by an order of magnitude.
- 07 Activation Outliers and Rotation-Based Quantisation A handful of channels in every transformer carry activations a hundred times larger than the rest, which is why activation quantisation fails where weight quantisation succeeds, and why rotating the hidden state fixes it.
- 08 Early Exit and Adaptive Depth Not every token needs all eighty layers, and the methods that act on this observation break batching and the KV cache in ways that decide whether the idea survives contact with a serving stack.
- 09 FlashAttention An IO-aware attention kernel that is both faster and lower-memory than the textbook implementation by tiling computation to keep activations in SRAM.
- 10 KV Cache Eviction and Compression Once the KV cache outgrows HBM the only remaining lever is to keep less of it, and the choice between dropping tokens, quantising them, or skipping them per query decides which capability you lose first.
- 11 Mixture-of-Experts Inference Why serving MoE models is harder than serving dense models of equivalent quality, and how DeepSeek and Mistral made it work in production.
- 12 Paged Attention and the KV Memory Manager How vLLM's PagedAttention treats the KV cache like OS virtual memory, cutting fragmentation waste from 60-80% to under 4% and roughly doubling to quadrupling serving throughput.
- 13 Speculative Decoding Use a small draft model to propose tokens that a large verifier accepts or rejects in parallel, giving lossless 2-3x latency wins on autoregressive generation.
- 14 Ternary and Extreme Low-Bit Models Below about three bits, quantising a trained model stops working and you have to train in low precision from the start, which changes the arithmetic of inference from multiply-accumulate to add-subtract.