Inference Optimisation
KV cache, FlashAttention, speculative decoding, quantisation and continuous batching.
7concepts
33flashcards
60minutes of reading
- 01 KV Cache Why decoder inference is quadratic without a KV cache and linear with one, and why managing that cache is now the dominant memory problem in LLM serving.
- 02 Quantisation - INT8, INT4, FP8 How to cut weight and activation precision below 16 bits without wrecking quality, and which scheme to pick for which deployment.
- 03 vLLM and Continuous Batching Why static batching wastes most of your GPU on variable-length workloads, and how iteration-level scheduling combined with PagedAttention raises throughput by an order of magnitude.