Sparsity & Pruning
Magnitude and second-order criteria, structured versus unstructured sparsity, and the hardware that rewards it.
11concepts
100flashcards
81minutes of reading
- 01 Activation Sparsity in Feedforward Layers Trained transformers fire only a few percent of their MLP neurons on any given token, which is a property of the activations rather than the weights, and it is the reason a dense model can be computed sparsely without deleting anything.
- 02 Magnitude Pruning and the Lottery Ticket Hypothesis Why the simplest possible pruning criterion is so hard to beat, what the lottery ticket experiment actually claims, and the rewinding detail that decides whether it reproduces.
- 03 Structured Versus Unstructured Sparsity Why 90 percent unstructured sparsity can be slower than a dense matmul, what removing a whole channel buys instead, and how to decide which side of the tradeoff a deployment sits on.
- 04 Cashing In Activation Sparsity on Real Hardware Fifty percent activation sparsity converts to roughly fifteen percent latency and a 2.5x FLOP cut converts to 1.40x on a GPU, because decode is bandwidth-bound, gathers are not free, and batching makes the union of active neurons dense.
- 05 Contextual Sparsity and Activation Predictors If you knew in advance which neurons a token will activate you could avoid loading the rest, so the whole technique reduces to building a predictor that is cheaper than the work it saves and almost never wrong.
- 06 Depth Pruning and Layer Removal for LLMs Deleting whole transformer blocks is the one form of structured pruning that speeds up batch-one decoding almost linearly, and the evidence says deep layers are surprisingly removable for knowledge recall and surprisingly necessary for reasoning.
- 07 Dynamic Sparse Training Training a sparse network from scratch by continuously rewiring which weights exist, why the gradients of absent weights are the key signal, and what stops this from replacing dense training.
- 08 N:M Semi-Structured Sparsity The compromise pattern that hardware can accelerate, why 2:4 specifically, and the gap between the theoretical 2x and what a full model actually achieves.
- 09 Relufication and Sparsity-Aware Pretraining Swapping SiLU back to ReLU and continuing to pretrain recovers 90 percent activation sparsity that SwiGLU models never had, and the explicit top-K and dReLU variants show the quality cost of that sparsity is close to zero if you train for it.
- 10 Second-Order and Activation-Aware Pruning How the Hessian of the loss gives a principled importance score, why the exact version is intractable, and the two approximations that made one-shot pruning of large language models work.
- 11 Training-Free Activation Sparsification A magnitude threshold calibrated per layer gets 40 to 50 percent model-wide activation sparsity out of an off-the-shelf SwiGLU model with no retraining and no predictor network, which is a smaller win than relufication and costs nothing to try.