Mixed-Precision Bit Allocation and Sensitivity

A "Q4_K_M" quantised Llama file is not a 4-bit model. When llama.cpp introduced its k-quant formats in June 2023, that mix stored most tensors in a 4.5 bits-per-weight format but used the 6.5625-bit format for half of the attention value projections and the feed-forward output projections (llama.cpp PR #1684, k-quants). Someone decided those tensors were more fragile than the rest and spent extra bits on them by hand. Mixed-precision allocation is the attempt to make that decision from measurement instead of intuition.

GPTQ and AWQ, the neighbouring concepts, decide how to round the weights inside one layer at a fixed bit width. This concept is about the layer above: given a memory or latency budget, which layers get which bit width.

An exponential search with a second-order shortcut

With \(L\) quantisable tensors and \(|\mathcal{B}|\) candidate bit widths, there are \(|\mathcal{B}|^L\) assignments. A 32-block transformer with seven linear projections per block has 224 tensors, so three candidate widths already give \(3^{224}\) configurations. Evaluating each one is out of the question.

The shortcut is a Taylor expansion of the loss around the converged full-precision weights. The gradient is approximately zero there, so a quantisation perturbation \(\Delta W_i = Q(W_i) - W_i\) to layer \(i\) changes the loss by roughly

where \(H_i\) is the Hessian of the loss with respect to that layer's weights. Layers in sharp regions of the loss surface are expensive to perturb; layers in flat regions are cheap.

HAWQ ranked layers by the top Hessian eigenvalue (Dong et al., 2019, HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision, arXiv:1905.03696). Its successor argued that one eigenvalue is the wrong summary, with a two-line example: \(F_1 = 100x^2 + y^2\) and \(F_2 = 100x^2 + 99y^2\) share a top eigenvalue of 200, yet \(F_2\) is far more sensitive along \(y\). HAWQ-V2 instead scores a candidate assignment by

with \(\overline{\mathrm{Tr}}(H_i)\) the average Hessian trace per parameter (Dong et al., 2020, HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks, arXiv:1911.03852). The trace never requires forming \(H\): Hutchinson's estimator uses \(\mathrm{Tr}(H) = \mathbb{E}[z^\top H z]\) for random Rademacher or Gaussian \(z\), and each \(z^\top H z\) costs one Hessian-vector product. The paper reports traces for all 54 layers of ResNet-50 in 30 minutes on four GPUs.

From scores to an allocation

Once each layer has a sensitivity and each bit width a perturbation, allocation is a knapsack. HAWQ-V3 writes it as an integer linear programme: minimise \(\sum_i \Omega_i(b_i)\) subject to constraints on total model size, bit operations, or measured latency, solvable in seconds (Yao et al., 2021, HAWQ-V3: Dyadic Neural Network Quantization, arXiv:2011.10680). On ResNet-50 its INT4/INT8 mix cut latency 23% below INT8 while keeping 76.73% top-1, against 77.58% for pure INT8.

A toy version shows why uneven allocation wins. Uniform quantisation error per weight scales roughly as \(4^{-b}\), since each extra bit halves the step. Take two equal-sized layers with per-parameter sensitivities 10 and 1 and a budget of 3.5 bits on average. Giving the sensitive layer 4 bits and the other 3 costs \(10\cdot4^{-4} + 1\cdot4^{-3} \approx 0.055\). Reversing the assignment costs \(10\cdot4^{-3} + 4^{-4} \approx 0.160\), nearly three times worse for the same memory.

The same logic carried to transformers early. Q-BERT used Hessian information to assign mixed precision within BERT and reported weights compressed 13 times, down to 2 bits in places, with at most 2.3% accuracy loss on its four tasks (Shen et al., 2020, Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT, arXiv:1909.05840).

When it breaks

The proxy and the hardware can disagree. HAQ rejected sensitivity proxies and FLOPs altogether, training a reinforcement-learning agent on direct latency and energy feedback from a hardware simulator, and found that the best policies for edge and cloud accelerators differed (Wang et al., 2019, HAQ: Hardware-Aware Automated Quantization with Mixed Precision, arXiv:1811.08886). A Hessian ranking says nothing about whether a 3-bit kernel exists or is faster than a 4-bit one on the target device.

Additivity is assumed, not measured. \(\Omega\) sums per-layer terms and ignores cross-layer Hessian blocks. Quantising two adjacent layers can hurt more than the sum of quantising each, and the error of the first shifts the activations the second sees.

Full Hessians do not scale to LLMs. Trace estimation needs second-order backward passes through the model. A common fallback for LLMs is the Fisher approximation \(H \approx \frac{1}{|D|}\sum_d g_d g_d^\top\), often diagonal, as SqueezeLLM does for its sensitivity weighting (Kim et al., 2024, SqueezeLLM: Dense-and-Sparse Quantization, arXiv:2306.07629). That approximation is only as good as the gradients from a small calibration set.

Heterogeneity has a systems cost. Every distinct bit width needs a kernel, and mixed layouts complicate batching and memory planning. A plausible reason hand-written recipes like the k-quant mixes remain common is that a few fixed layouts are easy to ship and test, even where a measured allocation might lose less accuracy.