Sparsity & Pruning
Magnitude and second-order criteria, structured versus unstructured sparsity, and the hardware that rewards it.
5concepts
62flashcards
36minutes of reading
- 01 Magnitude Pruning and the Lottery Ticket Hypothesis Why the simplest possible pruning criterion is so hard to beat, what the lottery ticket experiment actually claims, and the rewinding detail that decides whether it reproduces.
- 02 Structured Versus Unstructured Sparsity Why 90 percent unstructured sparsity can be slower than a dense matmul, what removing a whole channel buys instead, and how to decide which side of the tradeoff a deployment sits on.
- 03 Dynamic Sparse Training Training a sparse network from scratch by continuously rewiring which weights exist, why the gradients of absent weights are the key signal, and what stops this from replacing dense training.
- 04 N:M Semi-Structured Sparsity The compromise pattern that hardware can accelerate, why 2:4 specifically, and the gap between the theoretical 2x and what a full model actually achieves.
- 05 Second-Order and Activation-Aware Pruning How the Hessian of the loss gives a principled importance score, why the exact version is intractable, and the two approximations that made one-shot pruning of large language models work.