GPTQ and Second-Order Weight Rounding

\(H = 2XX^\top\), the Hessian of the layer reconstruction error \(\lVert WX - \hat{W}X\rVert_2^2\) with respect to the weights. It is a \(d_{\text{in}} \times d_{\text{in}}\) matrix built once per layer from a few hundred calibration sequences, and it is shared across all output rows, which is what makes the method affordable.

It distributes the error just introduced across the remaining unquantised weights, in proportion to how correlated their inputs are with the quantised one, using a closed-form update scaled by the inverse Hessian. The point is that output error, unlike weight error, can be partly cancelled by adjusting a different weight.

Fixed left-to-right quantisation order instead of greedy selection, so one Hessian inverse serves every row; all output rows processed simultaneously since they share \(H\), turning scalar updates into matrix operations; and lazy block updates in groups of 128 columns to keep the working set in cache. Together these bring a 175B model to a few GPU-hours.

Because the entire compensation is computed from \(H = 2XX^\top\), which encodes that specific input distribution. Calibrating on English and serving multilingual traffic optimises the rounding for correlations that do not hold at deployment, and calibration-set perplexity will not reveal it. Round-to-nearest has no such dependence because it ignores inputs entirely.

Error propagation. Each layer is quantised against the original model's activations, so quantisation error accumulated in earlier layers never enters the objective. Sequential variants feed each layer the already-quantised model's activations instead, which is meaningfully better at low bit width and costs one extra forward pass through the partially quantised model.

A near-singular Hessian, which arises whenever the calibration set is small relative to the input dimension, making \(H^{-1}\) unstable. Adding about one percent of the mean diagonal keeps the inverse finite. Too little damping and the compensation explodes; too much and the method degenerates toward plain round-to-nearest.

\(H = 2XX^\top\), the Hessian of the layer reconstruction error \(\lVert WX - \hat{W}X\rVert_2^2\) with respect to the weights. It is a \(d_{\text{in}} \times d_{\text{in}}\) matrix built once per layer from a few hundred calibration sequences, and it is shared across all output rows, which is what makes the method affordable.

It distributes the error just introduced across the remaining unquantised weights, in proportion to how correlated their inputs are with the quantised one, using a closed-form update scaled by the inverse Hessian. The point is that output error, unlike weight error, can be partly cancelled by adjusting a different weight.

Fixed left-to-right quantisation order instead of greedy selection, so one Hessian inverse serves every row; all output rows processed simultaneously since they share \(H\), turning scalar updates into matrix operations; and lazy block updates in groups of 128 columns to keep the working set in cache. Together these bring a 175B model to a few GPU-hours.

Because the entire compensation is computed from \(H = 2XX^\top\), which encodes that specific input distribution. Calibrating on English and serving multilingual traffic optimises the rounding for correlations that do not hold at deployment, and calibration-set perplexity will not reveal it. Round-to-nearest has no such dependence because it ignores inputs entirely.

Error propagation. Each layer is quantised against the original model's activations, so quantisation error accumulated in earlier layers never enters the objective. Sequential variants feed each layer the already-quantised model's activations instead, which is meaningfully better at low bit width and costs one extra forward pass through the partially quantised model.

A near-singular Hessian, which arises whenever the calibration set is small relative to the input dimension, making \(H^{-1}\) unstable. Adding about one percent of the mean diagonal keeps the inverse finite. Too little damping and the compensation explodes; too much and the method degenerates toward plain round-to-nearest.