Activation Outliers and Rotation-Based Quantisation

Because past a few billion parameters transformers develop systematic outlier channels: a small set of hidden dimensions whose activations are one to two orders of magnitude larger than the rest, consistently across inputs.

The scale factor must cover the outlier's range, so the other 99.9% of values are squeezed into a handful of the 256 available levels and effective precision collapses to two or three bits.

Weight distributions have no such structure, which is exactly why weight-only quantisation succeeds where activation quantisation fails.

For \(Y = XW\), insert a diagonal \(s\):

The product is unchanged. Choosing \(s\) large on outlier channels shrinks those activations and grows the matching weight rows, moving difficulty from the side with no headroom to the side that has some.

It is free at runtime because the weight side is folded once offline and the activation side folds into the preceding LayerNorm's scale parameters. Reported: W8A8 at up to 1.56x speedup and 2x memory reduction.

A diagonal rescale only moves magnitude between the two sides of one matmul. The outlier still exists; it has just been shifted onto the weights. At 4 bits neither side has enough headroom to hide it.

A rotation changes the basis: \(XW = (XQ)(Q^{\top}W)\) with \(Q\) orthogonal. A randomised Hadamard \(Q\) spreads the outlier's energy across all dimensions, so \(XQ\) is approximately isotropic and has no outlier channels at all.

Because \(Q\) is orthogonal it can often be folded into adjacent weights, costing nothing.

QuaRot applied Hadamard rotations throughout the network so that all matmuls run in 4 bits, weights, activations and KV cache included, with no channels held back in higher precision. LLaMA2-70B: at most 0.47 WikiText-2 perplexity loss, 99% of zero-shot performance, and lossless 6- and 8-bit variants needing no calibration data.

SpinQuant learned the rotations on the Stiefel manifold instead of sampling them randomly: a 2.9-point gap to full precision on LLaMA-2 7B zero-shot reasoning, ahead of SmoothQuant by 25.0 and LLM-QAT by 19.1 points.

Because weight-only shrinks memory traffic but leaves the arithmetic in bf16. Quantising activations lets the matmul itself run on int8 or int4 tensor cores, which is where the raw FLOPS are.

That matters in the two regimes weight-only cannot help: prefill, which is compute bound rather than bandwidth bound, and large-batch serving, where the weight read is amortised across sequences and arithmetic dominates again.

Full 4-bit also quantises the KV cache, which is what actually caps long-context batch size.

Because outlier channels and the calibration or learned rotation that neutralises them are properties of the specific trained weights. Merging an adapter changes those weights, so the outlier structure can move and the transformation no longer matches what it was fitted to.

The result is silent quality loss: nothing errors, the checkpoint loads, and the quantisation is simply wrong for the model it is now applied to.

Any pipeline that merges adapters should requantise and re-gate automatically rather than relying on someone remembering.

Because past a few billion parameters transformers develop systematic outlier channels: a small set of hidden dimensions whose activations are one to two orders of magnitude larger than the rest, consistently across inputs.

The scale factor must cover the outlier's range, so the other 99.9% of values are squeezed into a handful of the 256 available levels and effective precision collapses to two or three bits.

Weight distributions have no such structure, which is exactly why weight-only quantisation succeeds where activation quantisation fails.

For \(Y = XW\), insert a diagonal \(s\):

The product is unchanged. Choosing \(s\) large on outlier channels shrinks those activations and grows the matching weight rows, moving difficulty from the side with no headroom to the side that has some.

It is free at runtime because the weight side is folded once offline and the activation side folds into the preceding LayerNorm's scale parameters. Reported: W8A8 at up to 1.56x speedup and 2x memory reduction.

A diagonal rescale only moves magnitude between the two sides of one matmul. The outlier still exists; it has just been shifted onto the weights. At 4 bits neither side has enough headroom to hide it.

A rotation changes the basis: \(XW = (XQ)(Q^{\top}W)\) with \(Q\) orthogonal. A randomised Hadamard \(Q\) spreads the outlier's energy across all dimensions, so \(XQ\) is approximately isotropic and has no outlier channels at all.

Because \(Q\) is orthogonal it can often be folded into adjacent weights, costing nothing.

QuaRot applied Hadamard rotations throughout the network so that all matmuls run in 4 bits, weights, activations and KV cache included, with no channels held back in higher precision. LLaMA2-70B: at most 0.47 WikiText-2 perplexity loss, 99% of zero-shot performance, and lossless 6- and 8-bit variants needing no calibration data.

SpinQuant learned the rotations on the Stiefel manifold instead of sampling them randomly: a 2.9-point gap to full precision on LLaMA-2 7B zero-shot reasoning, ahead of SmoothQuant by 25.0 and LLM-QAT by 19.1 points.

Because weight-only shrinks memory traffic but leaves the arithmetic in bf16. Quantising activations lets the matmul itself run on int8 or int4 tensor cores, which is where the raw FLOPS are.

That matters in the two regimes weight-only cannot help: prefill, which is compute bound rather than bandwidth bound, and large-batch serving, where the weight read is amortised across sequences and arithmetic dominates again.

Full 4-bit also quantises the KV cache, which is what actually caps long-context batch size.

Because outlier channels and the calibration or learned rotation that neutralises them are properties of the specific trained weights. Merging an adapter changes those weights, so the outlier structure can move and the transformation no longer matches what it was fitted to.

The result is silent quality loss: nothing errors, the checkpoint loads, and the quantisation is simply wrong for the model it is now applied to.

Any pipeline that merges adapters should requantise and re-gate automatically rather than relying on someone remembering.