Portfolio ↗

Knowledge Distillation

Soft targets and temperature, sequence-level and on-policy distillation, and when a student beats its teacher.

5concepts
64flashcards
36minutes of reading
Start with Temperature and Dark Knowledge Drill the deck
All levels beginner intermediate advanced
  1. 01 Temperature and Dark Knowledge What information a teacher's full output distribution carries that a hard label does not, why temperature is needed to expose it, and the gradient-scaling correction that everyone forgets. intermediate 7m 12 cards