Repeating Data and Data-Constrained Scaling

Compute-optimal scaling assumes fresh tokens are always available: for a given budget, spend it on parameters and data in proportion (Hoffmann et al., 2022, Training Compute-Optimal Large Language Models, arXiv:2203.15556). When unique data runs out before compute does, that recipe stops being actionable and the question becomes what a repeated token is worth relative to a new one.

The geometric-decay model

Muennighoff and colleagues ran roughly 400 training runs, up to 9 billion parameters and 900 billion tokens, to answer it, and their headline finding is unusually clean: training with up to four epochs of repeated data produces negligible change in loss compared with the same number of unique tokens, while beyond that the value of additional compute decays towards zero (Muennighoff et al., 2023, Scaling Data-Constrained Language Models, NeurIPS 2023, arXiv:2305.16264).

The fit that carries the result treats each pass as worth a constant fraction less than the one before. With \(U\) unique tokens, \(E\) epochs and a per-repetition decay \(\delta\), the effective unique data is

and the paper summarises the decay as \(R_D^{*} = (1-\delta)/\delta\), the scale at which repeated tokens stop contributing. Substituting \(D'\) for \(D\) in the Chinchilla loss form gives a scaling law that is usable when data is the binding constraint, and under it additional compute is best spent on both more epochs and more parameters rather than either alone.

Two consequences follow from the shape of the sum rather than from any fitted constant. Early epochs are nearly free, because \((1-\delta)^{k-1}\) is close to one for small \(k\). And the sum is bounded above by \(U/\delta\), so no amount of repetition substitutes for unique data in the limit.

Aggregate safety is not per-document safety

The four-epoch result describes repeating a whole corpus uniformly. Repeating part of it is a different and much worse operation. Anthropic's study of repeated data found that an 800 million parameter model could be degraded to the quality of a model half its size by repeating just 0.1% of its training data 100 times, with the rest left unique (Hernandez et al., 2022, Scaling Laws and Interpretability of Learning from Repeated Data, arXiv:2205.10487).

The same work reports a pronounced double descent in which test loss rises part-way through training, and interpretability evidence that the damage falls on copying and induction structures. The authors' reading is that an intermediate repetition range is the dangerous one, where memorising the repeated fraction consumes capacity that the model would otherwise spend on generalising circuits.

So two pipelines that both average four epochs can behave completely differently: one that repeats everything four times is roughly fine, while one that repeats most data once and a small high-quality shard hundreds of times is damaging itself in a way aggregate token counts do not reveal.

When it breaks

Repetition multiplies memorisation. Verbatim extraction risk scales with duplication count, so an epoch budget is also a privacy and copyright decision. Deduplication exists partly for this reason, and repeating a deduplicated corpus reintroduces the exposure that deduplication removed.

Contamination compounds. If evaluation text survived decontamination, every extra epoch raises the chance the model has effectively memorised the test set, and the benchmark improves for reasons that will not transfer.

The decay constant is not universal. It was fitted on specific models, data and benchmarks. Published numbers from other settings suggest architectures differ in repeat tolerance, and sparse models are reported to tolerate fewer passes than dense ones of the same effective capacity, which has not been independently established at scale. Treat four epochs as an anchor to ablate against, not a constant.

Loss is not the only objective. The negligible-change finding is about validation loss and aggregate benchmarks. Behaviours that depend on the tails of the distribution, such as rare factual recall, can degrade while the headline loss holds steady.