Structured Versus Unstructured Sparsity

A pruned model that is 90 percent zeros and runs at exactly the same speed as the dense original is the normal outcome, not a bug. The gap between a sparsity ratio and a speedup is the single most important thing to understand about pruning in practice, and it is a fact about memory systems rather than about arithmetic.

Why unstructured sparsity does not accelerate

Zeros stored in a dense tensor still occupy memory and still get multiplied. Realising the benefit means a sparse format such as CSR, which stores values plus indices. That immediately costs: at 90 percent sparsity with fp16 values and int32 indices, the index overhead is larger than the saved values, so the format only wins on memory above roughly 95 percent sparsity for that combination.

The bigger problem is execution. A dense matmul reads contiguous memory in a pattern the hardware prefetches perfectly and feeds into tensor cores that expect regular tiles. A sparse matmul reads indices, gathers scattered values, and produces irregular tiles that tensor cores cannot consume, so it falls back to slower general units. Sparse kernels typically need 95 to 99 percent sparsity before they beat a dense kernel on GPU, and even then only for specific shapes.

What structured pruning removes instead

Structured pruning removes whole units: an attention head, a filter, an MLP intermediate dimension, an entire layer. The result is a smaller dense model. A transformer with the MLP dimension cut from 11,008 to 8,192 is just a transformer with a smaller MLP, and it runs on unchanged kernels at proportionally lower cost.

The speedup is therefore real and immediate on any hardware, with no format, no index overhead, and no kernel work. The cost is accuracy: removing a whole channel removes the useful weights inside it along with the useless ones, so a given accuracy budget buys far less structured sparsity than unstructured. Roughly, a model tolerating 60 percent unstructured pruning might tolerate 20 to 30 percent structured pruning at similar quality.

Deciding between them

The question is what is scarce. If the constraint is memory footprint, for instance fitting a model on a device, unstructured sparsity with a compressed storage format is viable, since only the storage matters and the tensor can be expanded before compute. If the constraint is latency or throughput, structured pruning is usually the only thing that helps, unless the hardware has explicit support for a semi-structured pattern.

The middle ground, N:M semi-structured sparsity, exists precisely because both extremes are unsatisfying.

When it breaks

Structured pruning changes shapes, and shapes have alignment requirements. Cutting an MLP dimension to 8,133 because that is where the criterion landed produces a tensor whose inner dimension is not a multiple of the tile size, and the kernel pads it back up. The realised speedup is then smaller than the parameter reduction, and the fix is to constrain the pruning to multiples of 64 or 128 from the start.

Removing attention heads is not uniform across layers. Measured head importance varies enormously by depth, and pruning a fixed fraction per layer leaves capacity on the table. Allocating a global budget across layers does better and complicates the shape-alignment problem, since different layers then have different head counts.

Depth pruning and width pruning fail differently. Removing whole layers preserves per-layer capability and shortens the residual stream, which hurts tasks needing many sequential steps of computation, particularly multi-step reasoning. Removing width degrades every layer slightly. At equal parameter reduction, depth pruning usually shows better perplexity and worse reasoning, which is a difference perplexity alone will not surface.

Sparsity ratios in papers are frequently not speedups. A claimed 90 percent sparsity is a claim about zeros. Unless the paper reports wall-clock latency on named hardware with a named kernel, it is not a claim about speed, and treating it as one is the most common planning error in this area.