Quantisation
Post-training and quantisation-aware methods, outlier channels, GPTQ and AWQ, and low-bit arithmetic formats.
11concepts
104flashcards
80minutes of reading
- 01 Calibration Data for Post-Training Quantisation The few hundred kilobytes of text that decide how a 70B model rounds, why the default recipe of 128 random web sequences is a convention rather than a result, and what the evidence says about choosing better.
- 02 Quantisation Grids, Scale and Zero Point The affine map between floating point and integers, why granularity is the single most consequential choice, and how clipping and rounding errors trade against each other.
- 03 When Quantisation Does Not Make Inference Faster Weight-only quantisation buys bandwidth, not arithmetic, so its speedup is a function of batch size and phase; the arithmetic that predicts where the gain disappears, and why a kernel can be slower than fp16.