Small Language Models and the Overtraining Regime

Chinchilla established that for a fixed training compute budget, the best loss comes from a model of a particular size trained on roughly 20 tokens per parameter (Hoffmann et al., 2022, arXiv:2203.15556). Almost every deployed small model violates this by a wide margin, training a smaller model on far more tokens than the optimum prescribes. That is not a misunderstanding of the result. It is a correct response to a different objective.

The objective Chinchilla optimises, and the one you have

Chinchilla minimises loss for a fixed training budget. A production system pays for training once and for inference on every request, and for a widely used model the inference bill dwarfs the training bill within months.

Under that objective, the right question is: for a fixed inference cost, what is the best loss achievable? Inference cost is set by parameter count, so the model size is fixed by the budget and training tokens become the free variable. The answer is to train the small model for much longer than compute-optimality suggests, accepting a worse loss-per-training-FLOP in exchange for a better loss at that inference cost.

The numbers moved a long way. Llama 3's 8B model was trained on around 15 trillion tokens, close to 1,900 tokens per parameter, roughly a hundred times the Chinchilla ratio. Several later small models push further. The returns diminish, and they diminish slowly enough that this remains worth doing.

Why the returns persist

Loss keeps falling with more data because the model has not memorised the data; it is still extracting structure. The scaling law's data term does not saturate abruptly, it decays as a power law, so each doubling of tokens buys a smaller but non-zero improvement. There is a floor set by the model's capacity, and small models approach it slowly enough that trillions of extra tokens still help.

Data quality changes where that floor sits. The Phi series argued that curated and synthetic textbook-quality data lets a much smaller model reach capabilities associated with larger ones, which is a claim about moving the scaling curve rather than about moving along it. The claim is contested, principally because benchmark contamination is very hard to rule out when synthetic data is generated by a model that has seen the benchmarks, and the honest summary is that data quality clearly matters and the magnitude of the effect is disputed.

When it breaks

Repeating data stops helping and starts hurting. Once the unique data is exhausted, additional epochs give diminishing returns and eventually degrade the model. Published work on data-constrained scaling finds up to roughly four epochs to be nearly as good as fresh data, with sharp degradation beyond that (Muennighoff et al., 2023, arXiv:2305.16264). Overtraining is a data problem before it is a compute problem.

Training cost is real and is paid up front. A hundredfold overtrain of an 8B model costs more compute than a compute-optimal run for a much larger model. This is only rational when the inference volume is large enough to amortise it, which makes it a decision about expected deployment scale rather than about model quality.

Overtrained models are harder to compress. A model trained far past compute-optimality has less redundancy, since its capacity is more fully used, so it quantises and prunes worse than a compute-optimal model of the same size. The two efficiency strategies partly conflict, and stacking them delivers less than the product of their individual gains.

Benchmarks reward the wrong thing. Small models are heavily tuned against public benchmarks, and the gap between benchmark performance and behaviour on unfamiliar tasks is wider for them than for large models. A small model matching a large one on MMLU while failing on the same task phrased unusually is the standard observation, and it is why held-out and private evaluations matter more here than anywhere else.