Scaling Laws and the Chinchilla Correction

For most of deep learning's history, you could not say in advance what a bigger model would do; you trained it and found out. Scaling laws changed that. They are empirical power-law relationships, measured across many orders of magnitude, that let you predict a model's loss from three inputs: how much compute you spend, how many parameters it has, and how many tokens it sees. Once loss is predictable, the central question of frontier training becomes an optimisation problem: given a fixed compute budget, how should you split it between a bigger model and more data? The field got that answer badly wrong, then corrected it, and the correction reshaped how every lab spends its GPUs.

The Kaplan power laws

Kaplan et al. (2020) showed that test loss falls as a power law in model size, dataset size, and compute, with clean trends spanning more than seven orders of magnitude. Two findings drove practice. First, scale was smooth and predictable: no magic thresholds, just a straight line on a log-log plot. Second, and more provocatively, larger models were far more sample-efficient, so the compute-optimal move appeared to be training very large models on a relatively modest number of tokens and stopping well before convergence. The takeaway the field absorbed was blunt: spend your compute on parameters.

That conclusion drove a generation of models, GPT-3 at 175B, Gopher at 280B, Megatron-Turing at 530B, all enormous and, it turned out, trained on too little data.

The Chinchilla correction

Hoffmann et al. (2022), the Chinchilla paper, redid the analysis more carefully and reached the opposite operating point. Their headline rule: for compute-optimal training, model size and training tokens should scale in equal proportion. Every doubling of parameters should be matched by a doubling of training tokens. By the earlier Kaplan-era recipe, the giant models had far too many parameters for the tokens they saw.

The proof was direct. They trained Chinchilla, a 70B model, on roughly 4x more data than the 280B Gopher, holding the compute budget fixed. The smaller, data-richer model won across benchmarks, reaching 67.5% on MMLU, over seven points above Gopher, while being cheaper to fine-tune and far cheaper to serve. A rough rule of thumb fell out: around 20 training tokens per parameter is in the compute-optimal region.

Why the correction mattered beyond training

Chinchilla's logic has a second edge that the compute-optimal frame understates. Compute-optimal balances training cost only. But a model is trained once and served billions of times, so for anything deployed at scale it is rational to push past the Chinchilla-optimal token count: keep feeding data to a smaller model so that inference, the cost you pay forever, stays cheap. This is the explicit logic behind the Llama models, which are trained on far more tokens per parameter than 20:1 because the goal is a small model that is cheap to run, not a model that is cheap to train. Compute-optimal is the training answer; inference economics often argue for going further.

The data wall

Scaling laws assume you can always buy more of all three inputs. Compute and parameters are money. Tokens are not, indefinitely. High-quality public text is finite, and the largest training runs are now within striking distance of exhausting it. That pressure is the engine behind synthetic data, aggressive deduplication and quality filtering, and multi-epoch training, all attempts to keep the data axis growing once the easy tokens run out. The neat power law does not promise the supply that feeds it.

When it falls down

• Loss is not capability. Scaling laws predict next-token loss, not whether a model can do a specific task. The mapping from a lower loss to a benchmark gain is real but noisy, and downstream "emergent" jumps do not always track the smooth loss curve.
• Coefficients are setup-specific. The exponents and constants depend on architecture, tokeniser, data distribution, and optimiser. Borrowing another lab's fitted constants and trusting them precisely is a way to mis-budget a run.
• Optimal assumes one training run. The compute-optimal point ignores inference. For a model you will serve heavily, deliberately overtrain a smaller model (see the Llama logic above).
• The data axis can stall. Run out of high-quality tokens and adding parameters yields diminishing returns, no matter what the parameter power law alone suggests.

Further reading

• Scaling Laws for Neural Language Models - Kaplan et al.; the original power laws and the sample-efficiency argument.
• Training Compute-Optimal Large Language Models - the Chinchilla paper; the equal-scaling rule and the 70B-beats-280B demonstration.