Inference Optimisation
KV cache, FlashAttention, speculative decoding, quantisation and continuous batching.
14concepts
170flashcards
115minutes of reading
- 01 Evaluating Quantised Models A quantised model is meant to be a drop-in replacement, so matching aggregate accuracy is the wrong acceptance test; what you need to measure is how often its answers differ from the original's.
- 02 KV Cache Why decoder inference is quadratic without a KV cache and linear with one, and why managing that cache is now the dominant memory problem in LLM serving.
- 03 Quantisation - INT8, INT4, FP8 How to cut weight and activation precision below 16 bits without wrecking quality, and which scheme to pick for which deployment.
- 04 Structured Pruning and Sparsity Why removing half a model's weights is easy and making it run twice as fast is not, and how the field converged on layer, head, and channel removal followed by distillation.
- 05 Weight-Only Post-Training Quantisation Compressing weights to four bits while leaving activations in bf16 is the default way large models are made to fit, and GPTQ and AWQ get there by two genuinely different arguments about which errors matter.
- 06 vLLM and Continuous Batching Why static batching wastes most of your GPU on variable-length workloads, and how iteration-level scheduling combined with PagedAttention raises throughput by an order of magnitude.