Quantisation
Post-training and quantisation-aware methods, outlier channels, GPTQ and AWQ, and low-bit arithmetic formats.
5concepts
64flashcards
35minutes of reading
- 01 AWQ and Activation-Aware Scaling The observation that weight importance is determined by activation magnitude rather than weight magnitude, and how a per-channel rescaling protects the important weights without keeping any of them in higher precision.
- 02 GPTQ and Second-Order Weight Rounding Why rounding each weight to its nearest grid point is not the best rounding, how the Hessian of the layer reconstruction error tells you what to do instead, and what the approximations cost.
- 03 Low-Bit Number Formats and Microscaling Why floating-point formats at 8 bits and below split differently between exponent and mantissa, what a shared block exponent buys, and how hardware support decides which format wins regardless of its numerical merits.
- 04 Quantisation-Aware Training and the Straight-Through Estimator How you backpropagate through a step function that has zero gradient everywhere, what QAT buys over post-training methods, and why it is used far less than its accuracy would justify.