Sparsity & Pruning
Magnitude and second-order criteria, structured versus unstructured sparsity, and the hardware that rewards it.
11concepts
100flashcards
81minutes of reading
- 01 Activation Sparsity in Feedforward Layers Trained transformers fire only a few percent of their MLP neurons on any given token, which is a property of the activations rather than the weights, and it is the reason a dense model can be computed sparsely without deleting anything.
- 02 Magnitude Pruning and the Lottery Ticket Hypothesis Why the simplest possible pruning criterion is so hard to beat, what the lottery ticket experiment actually claims, and the rewinding detail that decides whether it reproduces.
- 03 Structured Versus Unstructured Sparsity Why 90 percent unstructured sparsity can be slower than a dense matmul, what removing a whole channel buys instead, and how to decide which side of the tradeoff a deployment sits on.