Dynamic Sparse Training
Training a sparse network from scratch by continuously rewiring which weights exist, why the gradients of absent weights are the key signal, and what stops this from replacing dense training.
Every pruning method described so far trains a dense model first. That means paying full dense training cost to produce a sparse one, which is fine when inference dominates the lifetime bill and unsatisfying when training is the expensive part. Dynamic sparse training starts sparse and stays sparse, changing which weights exist as it goes.
Fixed masks are not enough
Training with a random sparse mask fixed from initialisation performs poorly. The mask determines the model's hypothesis space, and a random choice made before seeing any data is almost certainly wrong. The network cannot recover, because a weight that is absent has no gradient and no way to come back.
Dynamic methods add a rewiring step. Periodically, drop some fraction of the current connections and grow an equal number elsewhere, keeping the parameter count fixed. Sparse Evolutionary Training drops the smallest-magnitude weights and grows at random. RigL (Evci et al., 2020, arXiv:1911.11134) keeps the magnitude-based drop and makes the growth informed.
The gradient of an absent weight
RigL's contribution is the observation that a weight which is zero still has a well-defined gradient. For a linear layer, \(\partial \mathcal{L} / \partial W_{ij} = \delta_i x_j\), which depends only on the incoming activation and the outgoing error signal, both of which exist whether or not \(W_{ij}\) does. So you can ask, for every absent connection, how useful it would be if it existed, and grow the ones with the largest magnitude gradient.
The catch is that computing this requires the dense gradient, which is a dense backward pass. RigL amortises it by rewiring infrequently, every few hundred steps, so the dense computation is a small fraction of total training cost. The rewiring rate is annealed to zero over training, so the topology settles and the final phase is ordinary sparse training on a fixed mask.
What it achieves and what it does not
RigL and its relatives match or beat iterative magnitude pruning at equal sparsity while never materialising a dense model, and they need fewer total FLOPs than train-then-prune. That is a genuine result about the training process.
The saving is theoretical on standard hardware. Training an 80 percent sparse network with unstructured masks does not run at 20 percent of dense cost, because the kernels do not exist, so implementations simulate sparsity with dense tensors and masks. The FLOP count falls; the wall clock does not. Realising the benefit requires either N:M-constrained dynamic sparsity, which hardware supports, or hardware built for unstructured sparse training, which is largely research.
When it breaks
Rewiring frequency is a delicate hyperparameter. Too frequent and the network never converges because its architecture keeps changing under the optimiser; too infrequent and it cannot escape a bad initial topology. Annealing the rate is standard and the schedule matters.
Optimiser state for a newly grown weight is undefined. A connection that reappears has no momentum or second-moment history. Initialising it to zero makes it move slowly and stay effectively absent; inheriting the pruned weight's state can inject stale information. Implementations differ here and the choice measurably affects results.
Very high sparsity disconnects the network. Above roughly 95 percent, layers can lose all paths between some inputs and outputs, and no rewiring rule that grows uniformly will reliably reconnect them. Per-layer sparsity budgets, typically keeping the first and last layers denser, are required rather than optional.
It has not transferred cleanly to large language models. The published results are largely on convolutional networks and moderate-scale training. At LLM scale the dominant costs are data and communication rather than dense FLOPs, the rewiring interacts badly with sharded optimiser states across data-parallel ranks, and the accuracy sensitivity is higher. Sparse pretraining of frontier models remains an open area rather than a settled technique.
14 flashcards for this concept
Click a card to reveal the answer.