Quantisation Grids, Scale and Zero Point

Weight distributions are roughly zero-centred, so a symmetric grid with \(z=0\) wastes nothing and keeps the integer matmul a plain multiply. Post-ReLU activations are non-negative, so a symmetric grid throws away half its levels. The cost of asymmetric is the cross term from expanding \((q_w - z_w)(q_a - z_a)\), which must be folded into the accumulator.

About 4.16 bits: 4 bits of weight plus one fp16 scale and one 4-bit zero point per 128 elements, which is roughly 0.16 bits amortised. At group size 32 the metadata rises to about 0.6 bits per weight, a sixth of the budget, which is why very small groups stop paying for themselves.

Because the scale factors out along the accumulation dimension: each output channel's scale multiplies its entire dot product, so it can be applied once after accumulation rather than per element. Per-group scales along the input dimension do not have this property, which is why they need kernel support rather than a post-hoc multiply.

Rounding error, bounded by \(s/2\) per element, shrinks as \(s\) shrinks. Clipping error, from values outside the representable range, grows as \(s\) shrinks. Taking \(s = \max\lvert x\rvert / q_{\max}\) eliminates clipping and is usually wrong, because a single outlier drags \(s\) up and imposes rounding error on everything else.

From roughly 6B parameters, specific hidden dimensions consistently carry values one to two orders of magnitude above the rest, across different inputs, and they matter for accuracy. That consistency is what makes them addressable: mixed-precision decomposition keeps those dimensions in fp16, and rotation methods spread their energy across dimensions before quantising. A random tail would allow neither.

Weight-only quantisation reduces bytes moved, not FLOPs. Decoding is memory-bandwidth-bound, so fewer bytes per weight translates directly into throughput. Prefill is compute-bound and the weights are dequantised to fp16 before the matmul anyway, so nothing changes. Unlocking compute speedup requires quantising activations too, at a much higher accuracy cost.

Perplexity averages per-token error over mostly easy tokens. Multi-step reasoning and code generation compound small per-token errors across a long output, so a model within a percent on perplexity can lose several points on those tasks. Evaluation has to cover the task class you actually serve, measured on realistic output lengths.

Because the objective is output error, not weight error, and output error is a quadratic form over the whole weight vector. Rounding one weight up creates an error a compensating change to a correlated weight can partly cancel. Nearest rounding is optimal per weight and blind to those interactions, which is why it collapses at 4 bits where compensating methods still work.

Weight distributions are roughly zero-centred, so a symmetric grid with \(z=0\) wastes nothing and keeps the integer matmul a plain multiply. Post-ReLU activations are non-negative, so a symmetric grid throws away half its levels. The cost of asymmetric is the cross term from expanding \((q_w - z_w)(q_a - z_a)\), which must be folded into the accumulator.

About 4.16 bits: 4 bits of weight plus one fp16 scale and one 4-bit zero point per 128 elements, which is roughly 0.16 bits amortised. At group size 32 the metadata rises to about 0.6 bits per weight, a sixth of the budget, which is why very small groups stop paying for themselves.

Because the scale factors out along the accumulation dimension: each output channel's scale multiplies its entire dot product, so it can be applied once after accumulation rather than per element. Per-group scales along the input dimension do not have this property, which is why they need kernel support rather than a post-hoc multiply.

Rounding error, bounded by \(s/2\) per element, shrinks as \(s\) shrinks. Clipping error, from values outside the representable range, grows as \(s\) shrinks. Taking \(s = \max\lvert x\rvert / q_{\max}\) eliminates clipping and is usually wrong, because a single outlier drags \(s\) up and imposes rounding error on everything else.

From roughly 6B parameters, specific hidden dimensions consistently carry values one to two orders of magnitude above the rest, across different inputs, and they matter for accuracy. That consistency is what makes them addressable: mixed-precision decomposition keeps those dimensions in fp16, and rotation methods spread their energy across dimensions before quantising. A random tail would allow neither.

Weight-only quantisation reduces bytes moved, not FLOPs. Decoding is memory-bandwidth-bound, so fewer bytes per weight translates directly into throughput. Prefill is compute-bound and the weights are dequantised to fp16 before the matmul anyway, so nothing changes. Unlocking compute speedup requires quantising activations too, at a much higher accuracy cost.

Perplexity averages per-token error over mostly easy tokens. Multi-step reasoning and code generation compound small per-token errors across a long output, so a model within a percent on perplexity can lose several points on those tasks. Evaluation has to cover the task class you actually serve, measured on realistic output lengths.

Because the objective is output error, not weight error, and output error is a quadratic form over the whole weight vector. Rounding one weight up creates an error a compensating change to a correlated weight can partly cancel. Nearest rounding is optimal per weight and blind to those interactions, which is why it collapses at 4 bits where compensating methods still work.