Weight-Only Post-Training Quantisation
Compressing weights to four bits while leaving activations in bf16 is the default way large models are made to fit, and GPTQ and AWQ get there by two genuinely different arguments about which errors matter.
Single-token decoding is memory-bandwidth bound, not compute bound. Every generated token requires reading the entire weight matrix from HBM and doing almost nothing with it: one matrix-vector product per layer. So the time to produce a token is roughly the time to stream the weights, and halving the bytes per weight nearly halves that time.
That is the whole economic argument for weight-only quantisation. Compute stays in bf16, activations stay in bf16, only the stored weights shrink, and they are dequantised on the fly inside the kernel. Nothing about the model's arithmetic changes; only what crosses the memory bus does.
The problem post-training quantisation solves
Round-to-nearest at 4 bits destroys a large model. The naive approach quantises each weight independently, minimising per-weight error, which is the wrong objective: what matters is the error in the layer output, and weights interact.
Both dominant methods therefore optimise a layer-level reconstruction objective. For a layer with weights \(W\) and calibration inputs \(X\), find quantised \(\hat{W}\) minimising
The two methods differ in how they attack it.
GPTQ: solve the reconstruction problem
GPTQ quantises weights one column at a time, and after fixing each column it updates all remaining unquantised weights to compensate for the error just introduced, using second-order information from the Hessian \(H = 2XX^{\top}\) of the reconstruction objective. The compensation is what makes it work: an error in column \(j\) is partly absorbed by columns \(j+1\) onward.
Reported results: a 175B model quantised to 3 or 4 bits in about four GPU hours, with negligible perplexity degradation, giving 3.25x inference speedup on an A100 and 4.5x on an A6000, and more than doubling the compression of prior one-shot methods (Frantar et al., 2022, GPTQ, ICLR 2023, arXiv:2210.17323).
AWQ: protect the weights that matter
AWQ starts from a different observation: weights are not equally important, and importance is visible in the activations, not the weights. Protecting roughly 1% of weight channels, those multiplied by large-magnitude activations, sharply reduces quantisation error.
Rather than keeping that 1% in higher precision, which is bad for kernels because it makes the layout ragged, AWQ scales those channels up before quantisation and scales the corresponding activations down by the same factor. The product is unchanged; the salient weights now occupy more of the quantisation grid. Every weight ends up at the same bit width, so the kernel stays uniform (Lin et al., 2023, AWQ, MLSys 2024, arXiv:2306.00978). The accompanying TinyChat runtime reported more than 3x speedup over FP16 on desktop and mobile GPUs.
The design choice that actually matters: group size
Both methods quantise in groups, with a shared scale (and often zero-point) per group of consecutive weights, commonly 128. Group size is the real quality knob.
Smaller groups track local weight distributions better and cost more metadata. At group size 128 with fp16 scale and zero-point, a 4-bit weight actually costs about 4.25 bits. At group size 32 it is closer to 5. "4-bit" in a model card is a marketing number; the effective bits per weight depends on grouping, and comparing two 4-bit checkpoints without comparing group size compares nothing.
When it breaks
Calibration data leaks into the result. Both methods need a calibration set, typically a few hundred sequences. Calibrate on WikiText and evaluate on WikiText and the numbers flatter. Calibrate on English and deploy in Japanese, or calibrate on prose and deploy on code, and quality drops in ways perplexity on the calibration domain will not show.
Weight-only quantisation does not help prefill. Prefill is compute bound, so shrinking weights buys little there and the dequantisation adds work. The speedup lives entirely in decode, which means the benefit depends on your input-to-output token ratio; a summarisation workload sees far less gain than a chat workload.
The KV cache is untouched. At long context the cache, not the weights, dominates memory. See KV cache eviction and compression.
Small models tolerate it worse. Quantisation error is absorbed by redundancy, and a 7B model has less of it than a 70B. The rule of thumb that 4-bit is free was established on large models and does not transfer down.
Perplexity is a weak acceptance test. See evaluating quantised models; a checkpoint can hold perplexity and lose materially on generation.
12 flashcards for this concept
Click a card to reveal the answer.