N:M Semi-Structured Sparsity

Unstructured sparsity gives the best accuracy per removed parameter and no speedup. Structured sparsity gives a guaranteed speedup and costs a lot of accuracy. N:M sparsity is a constrained pattern designed so that the hardware can decode it in fixed time while leaving enough freedom that accuracy holds up, and it is the only form of fine-grained sparsity with first-class support in current accelerators.

The pattern

In every contiguous group of \(M\) weights along the input dimension, at most \(N\) may be non-zero. The pattern shipped in NVIDIA Ampere and later is 2:4: exactly two of every four consecutive weights survive.

The regularity is what makes it tractable in silicon. Each group of four needs two values plus a 2-bit index per surviving weight identifying its position, so the metadata is a fixed 2 bits per stored value with no variable-length encoding and no pointer chasing. The sparse tensor core reads the metadata, selects the matching activations, and performs a dense multiply of half the width. Decode is constant-time and the datapath stays regular, which is exactly what unstructured sparsity denies it.

Storage is 50 percent of dense plus the metadata, roughly 56 percent overall for fp16. The tensor core throughput doubles for the matmul itself.

What is actually achieved

The 2x figure is for the sparse matmul in isolation. End-to-end results are smaller for reasons that are entirely predictable.

Not every operation is a matmul. Normalisation, activations, attention softmax, elementwise ops and residual adds are unaffected, so Amdahl's law applies to whatever fraction of runtime the linear layers occupy.

Decode in a language model is memory-bandwidth-bound, not compute-bound. Halving the FLOPs of an operation that was waiting on memory does not help; what helps is the roughly 44 percent reduction in bytes moved, which is a real but different benefit. Prefill, which is compute-bound, sees more of the matmul speedup.

Reported end-to-end gains on transformer inference cluster around 1.2x to 1.5x rather than 2x, and the exact number depends heavily on batch size and sequence length.

Getting the accuracy back

One-shot 2:4 pruning by magnitude loses noticeable quality. Two things recover most of it.

Applying a second-order or activation-aware criterion within each group of four, rather than plain magnitude, matters because the constraint is local: the method only has to decide which two of four to keep, and getting that right is exactly what Wanda-style and SparseGPT-style scoring does well.

Sparse fine-tuning after masking recovers most of the remainder. The established recipe is to train dense, apply the 2:4 mask, then fine-tune with the mask fixed, which is cheaper than the alternative of maintaining the constraint throughout training.

When it breaks

It is a hardware contract, not a general format. The pattern is only fast where the sparse tensor core exists. On CPU, on most edge accelerators, and on older GPUs, a 2:4 model is a dense model with half its weights set to zero, running at dense speed with dense memory use unless the runtime compresses it separately.

Framework support is narrower than hardware support. Getting the acceleration requires the tensor to be in the compressed layout and the operation to dispatch to the sparse kernel, which depends on shapes, dtypes and library versions. It is entirely normal to have a correctly 2:4-pruned model that never touches a sparse kernel, and the only reliable check is measuring latency rather than inspecting the mask.

The constraint is per-group, so it forces bad local choices. If four consecutive weights are all important, two must go regardless. Unstructured pruning at the same overall ratio would have taken them from somewhere else. This is precisely the accuracy that 2:4 pays for its speed, and it is why the pattern degrades faster than unstructured sparsity as models get smaller and less redundant.

Composing with quantisation is not additive. A 2:4 sparse INT4 model does not deliver the product of both speedups. The metadata overhead is a larger fraction of a 4-bit value than of a 16-bit one, and the kernel support for the combination is much thinner than for either alone.