Structured Pruning and Sparsity

You can set 50% of a 175-billion-parameter model's weights to zero in one pass, without retraining, and lose almost nothing in perplexity (Frantar and Alistarh, 2023, SparseGPT, arXiv:2301.00774). You will also observe exactly zero speedup, because a matrix with scattered zeros is still a dense matrix as far as a GPU is concerned. This gap between mathematical sparsity and realised throughput is the whole subject.

Unstructured, semi-structured, structured

Unstructured pruning zeroes individual weights wherever they are least useful. It gives the best quality at a given sparsity and needs specialised sparse kernels to convert into speed; on dense tensor cores it gives none.

Semi-structured pruning imposes a pattern the hardware understands. The 2:4 pattern, two zeros in every contiguous group of four weights, is supported by sparse tensor cores from NVIDIA Ampere onwards, with a compressed format that stores half the values plus indices. The ceiling is roughly 2× on the matmul, and reaching it depends on the layer being compute-bound in the first place.

Structured pruning removes whole units: attention heads, FFN channels, embedding dimensions, entire layers. The result is a smaller dense model that runs faster on any hardware with no special kernels. It is also the most damaging per parameter removed, which is why it is paired with retraining.

Choosing what to remove

Magnitude alone is a weak criterion in transformers because activation scales vary wildly across channels. The two methods that anchor the modern literature both fix this.

SparseGPT frames pruning as layer-wise reconstruction: for each layer, choose a mask and then update the remaining weights to minimise the change in that layer's output on a small calibration set, solving the resulting problem efficiently enough to prune a 175B model in a few hours on one GPU. Wanda simplifies the idea to a scoring rule with no weight update at all, ranking each weight by \(|W_{ij}| \cdot \lVert X_j \rVert_2\), the magnitude times the norm of the corresponding input activation, compared within each output row (Sun et al., 2023, arXiv:2306.11695). Wanda is a few lines of code and competitive with SparseGPT, which is a useful reminder that the activation term, not the solver, was doing most of the work.

Depth pruning is cruder and surprisingly effective. Measuring the representational similarity between layer inputs and outputs shows deep layers in open-weight models are highly redundant, and dropping a contiguous block of them degrades question-answering benchmarks little until a large fraction is gone, provided the final layer is kept and a short healing fine-tune follows (Gromov et al., 2024, The Unreasonable Ineffectiveness of the Deeper Layers, arXiv:2403.17887).

Pruning plus distillation is the production recipe

The strongest published pipeline does not treat pruning as a one-shot compression step. It prunes depth, width, attention and MLP dimensions from a large trained model, then retrains the survivor with knowledge distillation from the original, using under 3% of the original training token budget. That is far cheaper than training the small model from scratch and produced the Minitron family from Nemotron and Llama parents (Muralidharan et al., 2024, Compact Language Models via Pruning and Knowledge Distillation, arXiv:2407.14679).

When it breaks

• Perplexity is a forgiving metric. A pruned model can hold its language-modelling loss and lose multi-step reasoning, instruction following, or a low-resource language. Evaluate the capabilities you shipped, not the loss you trained on.
• Calibration data leaks into the result. One-shot methods select masks against a few hundred sequences. If those come from one domain, you have pruned for that domain.
• Sparsity does not compose with quantisation for free. Both consume the same accuracy headroom, and 4-bit plus 50% sparse is usually worse than either alone at matched memory.
• The saving may be in the wrong place. Decode is memory-bandwidth-bound; pruning FLOPs from a bandwidth-bound phase changes nothing. Prune parameters if you need memory, and prune depth if you need latency.
• MoE models complicate the story. Expert pruning removes capacity the router still expects to reach, and pruning by average utilisation collapses exactly the rare-domain experts that justified the architecture.