Temperature and Dark Knowledge
What information a teacher's full output distribution carries that a hard label does not, why temperature is needed to expose it, and the gradient-scaling correction that everyone forgets.
A classifier looking at a photograph of a BMW assigns it 0.99 to "car", 0.008 to "truck", and \(10^{-9}\) to "carrot". The one-hot training label says only "car" and discards the rest. But the ratio between the truck probability and the carrot probability encodes something real: this model has learned that trucks resemble cars and carrots do not. Hinton called that residual structure dark knowledge, and distillation is the practice of training a student on it rather than on the labels.
Why a hard label is a poor target
A one-hot label is maximally confident and carries no information about how classes relate. Training on it forces the student to spend capacity driving 999 logits toward negative infinity, which is both wasteful and a worse fit to the truth, since some of those classes genuinely are more plausible than others.
The teacher's full distribution is a softer, more informative target. It effectively supplies a similarity structure over the label space for free, and it also regularises: a target of 0.99 rather than 1.0 prevents the student from becoming pathologically overconfident, which is roughly why label smoothing works and why distillation improves calibration as a side effect.
Temperature
The problem is that the informative part of the distribution is numerically invisible. With logits \(z_i\), the softmax at temperature \(T\) is
At \(T = 1\) the interesting ratios sit at \(10^{-8}\) against a dominant 0.99, and their contribution to a cross-entropy loss is negligible. Raising \(T\) divides all logits, compressing the distribution and lifting the small probabilities into a range where they carry gradient. Typical values are 2 to 5; as \(T \to \infty\) the distribution flattens to uniform and all information is lost, so there is an optimum in between.
The student is trained with the same temperature on its own logits, and the usual objective mixes distillation against the true labels:
The \(T^2\) factor
That \(T^2\) is not a tuning constant. Softening the distribution by \(T\) scales the gradients of the soft-target term by approximately \(1/T^2\), so without the correction the distillation term's contribution shrinks quadratically as you raise the temperature, and \(\alpha\) silently means something different at every \(T\). Multiplying by \(T^2\) restores comparable gradient magnitudes and makes \(\alpha\) a meaningful, transferable mixing weight. It is stated in the original paper (Hinton, Vinyals and Dean, 2015, arXiv:1503.02531) and omitted often enough in reimplementations that a distillation run whose behaviour changes drastically with temperature is worth checking for it first.
When it breaks
In language modelling the vocabulary is the label space, and it is enormous. Storing a teacher's full distribution over 128,000 tokens for every position of a large corpus is prohibitive, so implementations store top-k logits, typically 50 to 100, and renormalise. That truncation discards precisely the long tail that dark knowledge lives in, which is why sequence-level and on-policy variants displaced naive logit matching for LLMs.
Temperature interacts with the teacher's calibration. A poorly calibrated, overconfident teacher has already collapsed its distribution, so raising \(T\) recovers less structure than it appears to. Distilling from a temperature-scaled, calibration-corrected teacher is a cheap improvement that is rarely done.
The hard-label term is not always helpful. When the teacher is substantially better than the labels are clean, mixing in ground truth pulls the student toward label noise the teacher had already smoothed away. On noisy datasets, \(\alpha\) close to 1 outperforms the conventional balanced setting.
Matching outputs does not transfer the mechanism. A student can match a teacher's distribution on the training distribution while computing it in an entirely different way, and consequently generalise differently off-distribution. Distillation transfers behaviour on the data you distil over, and nothing beyond it, which is the source of most surprises when a distilled model meets unfamiliar input.
12 flashcards for this concept
Click a card to reveal the answer.