Sparsity & Pruning
Magnitude and second-order criteria, structured versus unstructured sparsity, and the hardware that rewards it.
5concepts
62flashcards
36minutes of reading
- 01 Dynamic Sparse Training Training a sparse network from scratch by continuously rewiring which weights exist, why the gradients of absent weights are the key signal, and what stops this from replacing dense training.
- 02 N:M Semi-Structured Sparsity The compromise pattern that hardware can accelerate, why 2:4 specifically, and the gap between the theoretical 2x and what a full model actually achieves.
- 03 Second-Order and Activation-Aware Pruning How the Hessian of the loss gives a principled importance score, why the exact version is intractable, and the two approximations that made one-shot pruning of large language models work.