Magnitude Pruning and the Lottery Ticket Hypothesis

Remove the weights with the smallest absolute value. That is the entire criterion, it has been the baseline since the late 1980s, and thirty years of more principled alternatives have mostly failed to beat it by margins that survive careful comparison. Understanding why is more useful than memorising the alternatives.

Why magnitude works

The justification is a first-order argument: removing a weight \(w\) perturbs the output by roughly \(w\) times the corresponding input activation, so small weights perturb less. The argument is weak, since it ignores the activation magnitude entirely and ignores interactions between removed weights, and yet it works. The reason appears to be that trained networks develop a heavy-tailed weight distribution in which a minority of weights carry most of the function, and any criterion correlated with that ordering finds most of the available compression.

The procedure that matters is iterative rather than one-shot. Prune 20 percent, fine-tune to recovery, prune 20 percent of what remains, and repeat. Iterative magnitude pruning reaches far higher sparsity at a given accuracy than removing the same fraction in one step, because each round lets the surviving weights absorb the removed ones' function before the next cut.

The lottery ticket claim

Frankle and Carbin (2019, arXiv:1803.03635) reported something stranger than compression. Take a trained, iteratively pruned network. Reset the surviving weights to their original initial values, keeping the mask. Train again from that point. The result matches or exceeds the dense network's accuracy in comparable or fewer steps.

The claim is that a randomly initialised dense network contains a sparse subnetwork which, trained in isolation from its original initialisation, reaches full accuracy. Dense training's job, on this account, is partly to find that subnetwork.

The control that makes it interesting: reinitialise the same mask with fresh random values and it trains badly. The mask alone is not sufficient, and the initialisation alone is not sufficient. The pairing is what matters, which is a genuinely surprising claim about optimisation.

Rewinding, and why the original result did not scale

The original protocol resets to initialisation, step zero. On ResNet-50 and larger, it fails: the tickets do not train to full accuracy. The fix is late rewinding, resetting to the weights at some early point in training, typically 0.1 to 7 percent of the way through, rather than to step zero (Frankle et al., 2020, arXiv:1903.01611).

This is not a technicality; it changes what the hypothesis says. Large networks pass through an early unstable phase, and the useful structure emerges only after it. What the mask is compatible with is a point on the early trajectory, not the initialisation. Learning-rate rewinding, which resets the schedule but keeps the trained weights, matches or beats weight rewinding while being simpler, which further weakens the "special initialisation" framing.

When it breaks

Finding a ticket costs more than training densely. Iterative magnitude pruning requires several full train-prune-retrain cycles, so the total compute exceeds one dense run by a large factor. Lottery tickets are a statement about what exists inside a dense network, not a cheaper way to train, and they are frequently cited as though they were the latter.

Global versus layerwise thresholds behave very differently. Ranking all weights together lets layers with naturally larger weights survive at the expense of others, and can prune a narrow layer to nothing. Per-layer ratios avoid the collapse and give up cross-layer allocation. Neither is universally right, and a global threshold with a per-layer floor is the usual compromise.

Magnitude ignores the input distribution. A weight can be large and multiply an activation that is almost always near zero. This is the gap activation-aware criteria exploit, and it is why they beat magnitude specifically on transformers, where activation magnitudes vary by orders of magnitude across channels.

Unstructured sparsity does not accelerate anything by itself. A 90 percent sparse weight matrix stored densely occupies the same memory and takes the same time to multiply. The result is a research finding about redundancy until it is paired with a format and a kernel that exploit it, and on most hardware that means giving up some of the sparsity for structure.