Self-Distillation and Born-Again Networks
The result that a student identical in architecture to its teacher often outperforms it, the competing explanations for why, and where the effect is genuinely useful rather than merely surprising.
Train a network. Then train a second network with the identical architecture, from a fresh initialisation, using the first network's outputs as targets. The second one is usually better. Nothing was added: no extra data, no extra parameters, no extra labels. The result is robust across architectures and datasets and it remains only partly explained, which makes it one of the more interesting empirical facts in the field.
The setup
Born-again networks (Furlanello et al., 2018, arXiv:1805.04770) established the protocol. Generation 1 trains on hard labels. Generation 2 trains on generation 1's soft outputs, optionally mixed with hard labels. Generation 3 trains on generation 2's, and so on. Accuracy typically improves for two or three generations and then plateaus or declines. Ensembling the generations improves further, which is a hint about the mechanism.
Competing explanations
Soft targets as regularisation. The teacher's distribution is a smoothed label that encodes inter-class similarity, so it acts like a data-dependent, learned form of label smoothing. It reduces overconfidence and improves calibration, and calibration improvements are consistently observed.
Instance-level reweighting. Under this account, decomposing the distillation loss shows the teacher's confidence on each example acts as a per-example weight. Examples the teacher finds easy contribute less; ambiguous ones contribute more. The student therefore trains on a curriculum implicitly built by the teacher, without anyone designing one.
Multi-view feature learning. A more recent account argues that data has multiple predictive features per class, a single network learns a random subset determined by initialisation, and the teacher's soft outputs encode which subset it found. The student learns both its own subset and the teacher's, which is functionally an ensemble compressed into one model. This explanation predicts that ensembling generations helps, that the effect saturates once the available views are exhausted, and that it is stronger on data with genuinely multiple predictive views, all of which match observation.
None of these fully explains the phenomenon on its own, and the honest position is that the effect is well established and its mechanism is still argued about.
Where it is used deliberately
Label noise. A teacher trained on noisy labels produces soft targets that have partly averaged the noise away, so the student trained on them can beat the teacher on clean test data. This is the clearest practical use, and it is why self-distillation appears in pipelines with weak or automated labelling.
Deep supervision within one model. Distilling deeper layers' predictions into shallower ones, sometimes called self-distillation within a network, trains early exits without a separate teacher and supports adaptive-depth inference.
Semi-supervised expansion. Noisy Student training, which reached state-of-the-art ImageNet results, is self-distillation with unlabelled data and injected noise: the teacher labels unlabelled images, the student trains on them with heavy augmentation, and the student becomes the next teacher. Here the student is also larger, so it is not pure self-distillation, and the unlabelled data is doing much of the work.
When it breaks
Gains are small and are often confounded. One to two points on a classification benchmark is typical, and a second training run with a different seed, a longer schedule, or a tuned learning rate can produce a similar gain for the same compute. Claims about self-distillation need the compute-matched baseline, and papers that omit it are hard to interpret.
Error amplification across generations. Any systematic mistake in generation \(n\) becomes a training target for generation \(n+1\), so biases sharpen rather than wash out. Running many generations degrades the model, and the plateau after two or three is the point where amplification balances the benefit.
It is not a route around data scarcity. Nothing new enters the pipeline. Self-distillation reorganises what the model already extracted; it cannot add information the data did not contain, and treating it as a substitute for more or better data is a misreading of the result.
In generative models the loop is dangerous. Repeatedly training on a model's own generated output, rather than on soft targets for real data, narrows the distribution and degrades quality across generations. The distinction between distilling soft targets on real inputs and training on synthetic outputs matters enormously here, and conflating them is how self-distillation gets blamed for model collapse it did not cause.
14 flashcards for this concept
Click a card to reveal the answer.