Token-Level Versus Sequence-Level Distillation
Why matching a teacher's per-token distributions is not the same as matching its outputs, and how training on teacher-generated sequences changes the objective from mode-covering to mode-seeking.
Apply classification distillation to a sequence model and you get word-level knowledge distillation: at each position, minimise the KL divergence between teacher and student next-token distributions, given the ground-truth prefix. It works and it under-delivers, and the reason is that the quantity you care about is a distribution over sequences while the quantity you matched is a product of conditionals evaluated on prefixes the student will never produce.
The two objectives
Token-level distillation minimises
over positions of ground-truth sequences. It is dense supervision, one signal per position, and it is mode-covering: forward KL penalises the student for assigning low probability anywhere the teacher assigns mass, so the student spreads itself to cover the teacher's full distribution.
Sequence-level distillation (Kim and Rush, 2016, arXiv:1606.07947) generates output sequences from the teacher, usually by beam search, and trains the student with ordinary cross-entropy on those generated sequences as though they were ground truth. There are no soft targets at all. The student is being trained to imitate what the teacher actually produces.
Why sequence-level usually wins
A student with less capacity than its teacher cannot represent the teacher's full distribution. Forced by forward KL to cover all of it, it spreads its mass and ends up assigning moderate probability to a broad region, so its own greedy or beam decoding lands somewhere mediocre. Trained instead on the teacher's actual outputs, it concentrates on the mode the teacher would have chosen, which is the only thing that matters at inference because that is what decoding selects.
This is the mode-covering versus mode-seeking distinction again, and it explains why sequence-level distillation is the standard in machine translation and why every practical LLM distillation pipeline includes a teacher-generation phase. The generated corpus is also implicitly simpler than natural text, since a beam search output is a high-probability, low-entropy sequence, and a smaller student learns a low-entropy target more easily.
Exposure bias, and the flaw both share
Token-level distillation conditions on ground-truth prefixes. Sequence-level conditions on teacher-generated prefixes. Neither conditions on what the student itself will produce at inference, so both leave a train-test mismatch: the student is never taught how to recover from its own mistakes, because it never sees them.
This is the gap on-policy distillation closes, by generating sequences from the student and asking the teacher to score them. The supervision then lands exactly on the states the student actually visits.
When it breaks
Sequence-level distillation throws away the soft targets. All the dark knowledge is gone; the student sees only hard tokens. Hybrid schemes that generate from the teacher and then apply token-level KL over those generated sequences keep both, and are strictly better than either alone at the cost of storing or recomputing teacher logits.
Teacher generation is expensive and it is a one-time cost. Producing a distillation corpus means running full inference over the entire dataset with a large model, often with beam search. That cost is paid once and it is substantial enough to dominate the project budget for a large corpus, which is what makes on-policy methods, needing generation every step, look expensive by comparison.
The generated corpus inherits every teacher bias. Beam search output is systematically shorter, more generic and less diverse than natural text, and the student learns that as its target. Length bias in distilled translation models is a well-documented instance, and it appears in the student even when the teacher's sampled output shows no such bias.
Distributional coverage collapses. A student trained only on teacher generations sees a narrow slice of the space and behaves poorly outside it. Mixing real data with generated data mitigates this, and the ratio is a genuine hyperparameter rather than a detail.
12 flashcards for this concept
Click a card to reveal the answer.