Depth Pruning and Layer Removal for LLMs

Llama-2-7B has 32 transformer blocks. Delete 9 of them, chosen by a one-line similarity score and with no retraining at all, and the model keeps about 86 percent of its average benchmark performance (Men et al., 2024, ShortGPT: Layers in Large Language Models are More Redundant Than You Expect, arXiv:2403.03853). For a single-user decode loop, where each generated token walks every block in sequence, that is close to a 28 percent cut in per-token compute.

That is the appeal, and it is why depth pruning deserves separate treatment from the width-oriented methods in structured versus unstructured sparsity and the weight-level criteria in second-order and activation-aware pruning. Those decide which weights inside a layer matter. Depth pruning decides which layers matter at all.

Why a block can vanish

A pre-norm transformer block updates a residual stream additively: \(x_{\ell+1} = x_\ell + f_\ell(x_\ell)\), where \(x_\ell \in \mathbb{R}^d\) is the hidden state entering block \(\ell\) and \(f_\ell\) is the attention plus MLP sublayer. If \(\lVert f_\ell(x_\ell) \rVert\) is small relative to \(\lVert x_\ell \rVert\), the block barely rotates the stream, and removing it hands the next block an input it has nearly seen before. The residual norm tends to grow with depth, so later updates are often small in relative terms.

ShortGPT turns this into a score. Block Influence for layer \(\ell\) is

where \(t\) indexes token positions and the expectation runs over a calibration set \(X\). Low BI means input and output point the same way. You sort, delete the lowest-scoring blocks, and ship.

Gromov and colleagues ask a slightly different question: which contiguous run of \(n\) blocks can go? They choose the starting layer

using the representation at the final token \(T\), remove blocks \(\ell^*\) through \(\ell^*+n-1\), then heal the seam with a small QLoRA fine-tune on one 40 GB A100 (Gromov et al., 2024, The Unreasonable Ineffectiveness of the Deeper Layers, arXiv:2403.17887). On Llama-2-70B, MMLU barely moves until somewhere between 45 and 55 percent of layers are gone, then falls abruptly to chance. The removed blocks sit deep in the network, though practitioners keep the final block, which writes into the unembedding space.

What depth removal costs, and where it shows

The same Gromov study contains the finding that should govern any deployment. GSM8K degrades from the first pruned layer, with no plateau. Question-answering benchmarks that reward recalling a stored fact tolerate large cuts; tasks that need many sequential steps of computation do not. ShortGPT reports the generative version of the same asymmetry: at about 25 percent removal, the 7B model's scores on generation tasks such as XSum drop to near zero while multiple-choice scores hold.

A plausible reading: multiple-choice scoring compares a few log-likelihoods and tolerates a perturbed stream, while free generation feeds each error back as input, and multi-step reasoning appears to use depth as serial compute that layer removal cuts directly.

Minitron: width against depth, with distillation

NVIDIA's Minitron work treats pruning as the first step of a training recipe rather than the whole method: prune the teacher, then retrain the student by distilling from the unpruned model on a small token budget (Muralidharan et al., 2024, Compact Language Models via Pruning and Knowledge Distillation, arXiv:2407.14679). Deriving 8B and 4B models from a 15B parent used up to 40 times fewer training tokens than training each from scratch.

The follow-up ran the controlled comparison on Llama 3.1 8B, pruning to 4B by depth in one arm and by width (hidden size, attention, MLP) in the other, each distilled on 94B tokens (Sreenivas et al., 2024, LLM Pruning and Distillation in Practice: The Minitron Approach, arXiv:2408.11796):

At equal parameter count the depth variant is 50 percent faster and less than half as good at grade-school maths. That is the tradeoff in one row.

Where the literature disagrees

ShortGPT argues that plain layer deletion beats more elaborate width-pruning methods, and Kim et al. reach a similar conclusion, adding that width pruning buys little speed at the small batch sizes typical of memory-constrained serving (Kim et al., 2024, Shortened LLaMA, arXiv:2402.02834). Minitron's controlled comparison favours width for accuracy. Both can be right: the depth advocates mostly measured perplexity and multiple-choice tasks with light or no retraining, while Minitron measured reasoning after heavy distillation.

When it breaks

Similarity scores are calibration-dependent. BI and angular distance are measured on some text distribution. A block that looks redundant on web prose may be doing the work for code, tool-call formatting or a low-resource language, and the score will not warn you.

Healing hides damage from the metric you healed on. A QLoRA pass on generic text restores perplexity quickly. It restores multi-step reasoning much less reliably, and Kim et al. found that continued pretraining clearly outperforms LoRA at severe ratios.

Speedups are shape-dependent. Depth pruning's advantage is largest at batch size one, where decode is a chain of sequential layer passes. At large server batch sizes, where width-pruned matrices also run faster, the gap narrows.

The deep-layer finding may be a training artefact. Gromov et al. offer two readings: current pretraining underuses deep parameters, or shallow layers carry most stored knowledge. If future training recipes use depth better, the free lunch shrinks.