Knowledge Distillation
Soft targets and temperature, sequence-level and on-policy distillation, and when a student beats its teacher.
11concepts
101flashcards
82minutes of reading
- 01 Cross-Tokeniser Distillation Why a KL divergence between a teacher and a student with different tokenisers is not even well defined, the two distinct mismatches involved, and how optimal-transport and alignment-based losses get around them.
- 02 Distilling Rerankers into Retrievers How a cross-encoder's ranking signal is moved into a bi-encoder's parameters so it is paid at index time, why the target is a margin rather than a score, and why hard negatives decide whether any of it transfers.
- 03 Draft-Model Distillation for Speculative Decoding Why a draft model's standalone quality is the wrong objective, how distillation targets the acceptance rate directly, and why the training data has to come from the draft rather than the target.
- 04 On-Policy Distillation and Exposure Bias Why a student trained only on teacher trajectories cannot recover from its own errors, how generating from the student fixes the state distribution, and what reverse KL buys and costs.
- 05 Self-Distillation and Born-Again Networks The result that a student identical in architecture to its teacher often outperforms it, the competing explanations for why, and where the effect is genuinely useful rather than merely surprising.
- 06 The Capacity Gap in Distillation Why a stronger teacher can produce a worse student, what the intermediate-teacher fix does, and how to reason about the ratio between teacher and student capacity.
- 07 Token-Level Versus Sequence-Level Distillation Why matching a teacher's per-token distributions is not the same as matching its outputs, and how training on teacher-generated sequences changes the objective from mode-covering to mode-seeking.