The Capacity Gap in Distillation

The natural assumption is that a better teacher gives a better student. It is false often enough to be a named phenomenon. Distilling a very large model into a very small one frequently produces a worse student than distilling from a mid-sized teacher, and the effect is reproducible across vision and language.

Why a stronger teacher can hurt

Three mechanisms, and they compound.

The target is unrepresentable. A large teacher's function is complex enough that the student's hypothesis space contains nothing close to it. Forced to fit an unreachable target, the student converges to a poor compromise, in the same way a linear model asked to fit a highly non-linear function does worse than one asked to fit a mildly non-linear one.

Confident teachers carry less dark knowledge. Larger models are typically sharper on the training distribution, so their soft targets approach one-hot and the inter-class structure that distillation exists to transfer is compressed toward nothing. Higher temperature recovers some of it and cannot recover what is not there.

The optimisation landscape is harder. Fitting a complex target function is a harder optimisation problem, and the student's limited capacity means the loss surface it must navigate has worse conditioning. Students distilled from very large teachers show slower convergence and higher sensitivity to learning rate.

Teacher assistants

The standard fix is to distil in stages: the large teacher trains a mid-sized assistant, which trains the small student. Each step spans a smaller capacity gap, so each target is closer to representable. Reported gains are consistent and modest, and the cost is training an extra model.

The framing that makes this predictable is to think in terms of the ratio of parameter counts rather than the absolute sizes. Gaps of roughly an order of magnitude are usually fine; two orders of magnitude is where the effect becomes reliable. A 70B teacher into a 7B student is routine; a 70B teacher into a 0.5B student generally wants an intermediate step.

What actually transfers

The capacity gap is not only about parameters. A student with a different architecture, tokeniser or context length faces a structural gap that no amount of capacity closes. Distilling a transformer into a state space model, or across tokenisers, requires alignment machinery, usually mapping distributions across vocabularies, and consistently transfers less than a same-family distillation of comparable size ratio.

Conversely, the gap matters much less when the student is initialised from the teacher. Pruning a large model and then distilling the parent into the pruned child works well at ratios that would fail from scratch, because the student begins inside a region of function space near the teacher rather than searching for it.

When it breaks

The effect is task-dependent and not universal. On easy tasks with well-separated classes there is little dark knowledge to lose and large teachers distil fine. The phenomenon is most reliable on tasks with rich inter-class structure and on generative objectives with large output spaces.

Sequence-level distillation reduces the problem. Because the student learns from teacher outputs rather than teacher distributions, an unrepresentable target distribution matters less; the target becomes a corpus, which any model can fit in principle. This is part of why LLM distillation practice moved toward generated data and away from logit matching, and it means capacity-gap results from the classification literature transfer only partly.

Assistant chains accumulate error. Each stage loses something, and a three-stage chain can end up behind a well-tuned direct distillation. The gains are real and small, and adding stages past two is usually not worth the compute and the additional pipeline complexity.

Measuring the gap by parameter count is crude. Effective capacity depends on depth, width, and how much of the model is active per token. A sparse mixture-of-experts teacher with 400B total and 30B active parameters behaves, for distillation purposes, much more like a 30B teacher, and treating it as a 400B one predicts a gap that is not there.