Second-Order and Activation-Aware Pruning

Because the gradient is approximately zero at a converged solution, leaving the quadratic term \(\frac{1}{2}\delta w^\top H \delta w\) as leading. On a model that is not converged, or pruned mid-training, the first-order term is not negligible and every saliency derived this way is wrong, with nothing in the formula signalling it.

Off-diagonal Hessian terms, which give both a corrected saliency \(w_q^2 / 2[H^{-1}]_{qq}\) and a closed-form update to the remaining weights that compensates for the removal. The compensation is what lets second-order methods prune one-shot: removing a weight while adjusting its correlated neighbours costs far less loss than removing it alone.

By replacing the full-loss objective with per-layer reconstruction error \(\lVert WX - \hat{W}X\rVert^2\), whose Hessian is \(2XX^\top\) of size \(d_{\text{in}} \times d_{\text{in}}\) from a few hundred calibration samples, shared across all output rows. It then prunes column by column with OBS compensation. Same machinery as GPTQ with a different target, so the two compose into one pass.

\(\lvert w_{ij}\rvert \cdot \lVert x_j\rVert_2\): weight magnitude times the input activation's norm over the calibration set. No Hessian, no inverse, no weight update, running in seconds. That it matches SparseGPT at 50 percent sparsity says most of what the Hessian contributes on transformers is the activation scale, and that ignoring activation magnitude is the dominant error in plain magnitude pruning.

Above roughly 50 to 60 percent unstructured sparsity on language models, where one-shot methods degrade sharply. Past that some fine-tuning is required and the central selling point disappears. Quoting a method's 50 percent result while planning for 80 percent sparsity is a common and expensive extrapolation.

Error compounding across layers. Each layer is pruned to minimise its own output error against the original model's inputs, so damage accumulated upstream never enters the objective. Sequential variants that feed each layer the already-pruned model's activations correct this and matter more as sparsity rises.

Because the gradient is approximately zero at a converged solution, leaving the quadratic term \(\frac{1}{2}\delta w^\top H \delta w\) as leading. On a model that is not converged, or pruned mid-training, the first-order term is not negligible and every saliency derived this way is wrong, with nothing in the formula signalling it.

Off-diagonal Hessian terms, which give both a corrected saliency \(w_q^2 / 2[H^{-1}]_{qq}\) and a closed-form update to the remaining weights that compensates for the removal. The compensation is what lets second-order methods prune one-shot: removing a weight while adjusting its correlated neighbours costs far less loss than removing it alone.

By replacing the full-loss objective with per-layer reconstruction error \(\lVert WX - \hat{W}X\rVert^2\), whose Hessian is \(2XX^\top\) of size \(d_{\text{in}} \times d_{\text{in}}\) from a few hundred calibration samples, shared across all output rows. It then prunes column by column with OBS compensation. Same machinery as GPTQ with a different target, so the two compose into one pass.

\(\lvert w_{ij}\rvert \cdot \lVert x_j\rVert_2\): weight magnitude times the input activation's norm over the calibration set. No Hessian, no inverse, no weight update, running in seconds. That it matches SparseGPT at 50 percent sparsity says most of what the Hessian contributes on transformers is the activation scale, and that ignoring activation magnitude is the dominant error in plain magnitude pruning.

Above roughly 50 to 60 percent unstructured sparsity on language models, where one-shot methods degrade sharply. Past that some fine-tuning is required and the central selling point disappears. Quoting a method's 50 percent result while planning for 80 percent sparsity is a common and expensive extrapolation.

Error compounding across layers. Each layer is pruned to minimise its own output error against the original model's inputs, so damage accumulated upstream never enters the objective. Sequential variants that feed each layer the already-pruned model's activations correct this and matter more as sparsity rises.