Knowledge Distillation
Soft targets and temperature, sequence-level and on-policy distillation, and when a student beats its teacher.
11concepts
101flashcards
82minutes of reading
- 01 Feature and Attention Transfer in Distillation How FitNets hints, attention transfer, TinyBERT and MiniLM supervise a student's intermediate representations rather than only its outputs, what alignment problem each design has to solve, and the evidence that feature matching can make a student worse.
- 02 Multi-Teacher and Ensemble Distillation Why compressing an ensemble was distillation's original purpose, how averaging teachers in probability space differs from averaging logits, and why teacher disagreement is the only part of the signal that is worth paying for.
- 03 Temperature and Dark Knowledge What information a teacher's full output distribution carries that a hard label does not, why temperature is needed to expose it, and the gradient-scaling correction that everyone forgets.
- 04 Transfer Sets and Data-Free Distillation Distillation needs inputs to query the teacher on but not labels, which makes the choice of transfer set the dominant design decision, and makes distillation possible even when the original training data is gone.
- 05 Cross-Tokeniser Distillation Why a KL divergence between a teacher and a student with different tokenisers is not even well defined, the two distinct mismatches involved, and how optimal-transport and alignment-based losses get around them.
- 06 Distilling Rerankers into Retrievers How a cross-encoder's ranking signal is moved into a bi-encoder's parameters so it is paid at index time, why the target is a margin rather than a score, and why hard negatives decide whether any of it transfers.
- 07 Draft-Model Distillation for Speculative Decoding Why a draft model's standalone quality is the wrong objective, how distillation targets the acceptance rate directly, and why the training data has to come from the draft rather than the target.
- 08 On-Policy Distillation and Exposure Bias Why a student trained only on teacher trajectories cannot recover from its own errors, how generating from the student fixes the state distribution, and what reverse KL buys and costs.
- 09 Self-Distillation and Born-Again Networks The result that a student identical in architecture to its teacher often outperforms it, the competing explanations for why, and where the effect is genuinely useful rather than merely surprising.
- 10 The Capacity Gap in Distillation Why a stronger teacher can produce a worse student, what the intermediate-teacher fix does, and how to reason about the ratio between teacher and student capacity.
- 11 Token-Level Versus Sequence-Level Distillation Why matching a teacher's per-token distributions is not the same as matching its outputs, and how training on teacher-generated sequences changes the objective from mode-covering to mode-seeking.