Knowledge Distillation
Soft targets and temperature, sequence-level and on-policy distillation, and when a student beats its teacher.
11concepts
101flashcards
82minutes of reading
- 01 Feature and Attention Transfer in Distillation How FitNets hints, attention transfer, TinyBERT and MiniLM supervise a student's intermediate representations rather than only its outputs, what alignment problem each design has to solve, and the evidence that feature matching can make a student worse.
- 02 Multi-Teacher and Ensemble Distillation Why compressing an ensemble was distillation's original purpose, how averaging teachers in probability space differs from averaging logits, and why teacher disagreement is the only part of the signal that is worth paying for.
- 03 Temperature and Dark Knowledge What information a teacher's full output distribution carries that a hard label does not, why temperature is needed to expose it, and the gradient-scaling correction that everyone forgets.
- 04 Transfer Sets and Data-Free Distillation Distillation needs inputs to query the teacher on but not labels, which makes the choice of transfer set the dominant design decision, and makes distillation possible even when the original training data is gone.