Knowledge Distillation
Soft targets and temperature, sequence-level and on-policy distillation, and when a student beats its teacher.
5concepts
64flashcards
36minutes of reading
- 01 On-Policy Distillation and Exposure Bias Why a student trained only on teacher trajectories cannot recover from its own errors, how generating from the student fixes the state distribution, and what reverse KL buys and costs.
- 02 Self-Distillation and Born-Again Networks The result that a student identical in architecture to its teacher often outperforms it, the competing explanations for why, and where the effect is genuinely useful rather than merely surprising.
- 03 The Capacity Gap in Distillation Why a stronger teacher can produce a worse student, what the intermediate-teacher fix does, and how to reason about the ratio between teacher and student capacity.
- 04 Token-Level Versus Sequence-Level Distillation Why matching a teacher's per-token distributions is not the same as matching its outputs, and how training on teacher-generated sequences changes the objective from mode-covering to mode-seeking.