Sparsity & Pruning
Magnitude and second-order criteria, structured versus unstructured sparsity, and the hardware that rewards it.
11concepts
100flashcards
81minutes of reading
- 01 Cashing In Activation Sparsity on Real Hardware Fifty percent activation sparsity converts to roughly fifteen percent latency and a 2.5x FLOP cut converts to 1.40x on a GPU, because decode is bandwidth-bound, gathers are not free, and batching makes the union of active neurons dense.
- 02 Contextual Sparsity and Activation Predictors If you knew in advance which neurons a token will activate you could avoid loading the rest, so the whole technique reduces to building a predictor that is cheaper than the work it saves and almost never wrong.
- 03 Depth Pruning and Layer Removal for LLMs Deleting whole transformer blocks is the one form of structured pruning that speeds up batch-one decoding almost linearly, and the evidence says deep layers are surprisingly removable for knowledge recall and surprisingly necessary for reasoning.
- 04 Dynamic Sparse Training Training a sparse network from scratch by continuously rewiring which weights exist, why the gradients of absent weights are the key signal, and what stops this from replacing dense training.
- 05 N:M Semi-Structured Sparsity The compromise pattern that hardware can accelerate, why 2:4 specifically, and the gap between the theoretical 2x and what a full model actually achieves.
- 06 Relufication and Sparsity-Aware Pretraining Swapping SiLU back to ReLU and continuing to pretrain recovers 90 percent activation sparsity that SwiGLU models never had, and the explicit top-K and dReLU variants show the quality cost of that sparsity is close to zero if you train for it.
- 07 Second-Order and Activation-Aware Pruning How the Hessian of the loss gives a principled importance score, why the exact version is intractable, and the two approximations that made one-shot pruning of large language models work.
- 08 Training-Free Activation Sparsification A magnitude threshold calibrated per layer gets 40 to 50 percent model-wide activation sparsity out of an off-the-shelf SwiGLU model with no retraining and no predictor network, which is a smaller win than relufication and costs nothing to try.