Model Collapse from Recursive Training

By the time GPT-4-class models had been deployed publicly, a substantial fraction of new text appearing on the internet was already generated by models trained on earlier internet text. Feed that new web corpus back into the next training run, and you have a closed loop. Shumailov et al. (2023) named the resulting degradation model collapse: "use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear."

The word "irreversible" is doing real work there. This is not a temporary calibration error you can anneal away; it is a structural loss of distributional knowledge that compounds with each generation.

What collapse actually destroys

A language model does not store facts as key-value entries. It encodes a probability distribution over sequences. The high-probability region, broadly, is fluent, common-sense text. The low-probability tails hold rare languages, specialist vocabulary, unusual but valid sentence constructions, minority viewpoints, and long-tail factual knowledge.

When you sample from generation-\(n\) and use those samples as training data for generation-\((n+1)\), you introduce a systematic bias: you over-represent whatever the model was already confident about, and under-represent everything near the tails. The next model learns from this biased corpus, becoming even more confident about the centre and even less aware of the periphery. Repeat.

Formally, let \(p_0\) be the true data distribution and \(\hat{p}_n\) the model's distribution at generation \(n\). Each generation trains on samples from \(\hat{p}_{n-1}\), introducing two compounding error sources:

1. Approximation error - the model at generation \(n-1\) is itself an imperfect estimate of \(p_0\).
2. Sampling error - drawing a finite sample from \(\hat{p}_{n-1}\) adds additional variance, which is worst in the tails.

The result is that the variance of \(\hat{p}_n\) shrinks each generation. In a Gaussian toy model, after \(k\) generations the estimated variance \(\hat{\sigma}^2_k \approx \hat{\sigma}^2_0 - k \cdot \delta\) where \(\delta > 0\) depends on the ratio of synthetic to real samples. The distribution collapses inward. This is not a metaphor; it is literally what the maths shows.

The three-phase phenomenology

Empirical studies across VAEs, Gaussian mixture models, and LLMs (Shumailov et al., 2023) consistently show three qualitative phases:

The boundary between phases depends on the fraction of synthetic data in each training run, the model's expressivity, and whether any fresh real data is mixed in. With 100% synthetic replacement, collapse reaches the late phase in as few as five generations for smaller models.

Why accumulation is the lever

Gerstgrasser et al. (2024) showed that the framing of "synthetic data is dangerous" is imprecise. The critical variable is whether you replace real data with synthetic data or accumulate both. Their result:

• Replacement (each generation trains only on outputs from the previous one): test error diverges, confirming collapse.
• Accumulation (each generation trains on outputs from all prior generations plus the original real corpus): test error remains bounded, with a finite upper bound independent of the number of iterations.

The intuition is straightforward. In the accumulation regime, the original real-data signal is always present and its weight relative to synthetic data never falls to zero. The approximation errors introduced by sampling do not compound because the real distribution acts as a corrective anchor.

A minimal pseudo-trace of the two regimes:

# Replacement (collapse)
data_t = sample(model_{t-1}, n=100_000)        # no real data
model_t = train(data_t)

# Accumulation (stable)
data_t = real_data + sample(model_{t-1}, n=100_000)
model_t = train(data_t)

The difference is one line, but the long-run consequence is the difference between a usable model and a degenerate one.

Self-consuming loops in practice

Several common training workflows are self-consuming loops in disguise.

Constitutional AI and RLAIF generate model responses, score them with a critic model, and filter or label them for the next supervised fine-tuning round. If the critic and the generator share weights (or are successive checkpoints of the same model), every round the training distribution drifts further from human-generated text.

Iterative rejection sampling for instruction following (a standard step in Llama and Mistral fine-tuning pipelines) samples many completions, keeps those scoring above a threshold, and trains on the survivors. If the scoring model is itself the model being improved, the threshold is a moving target anchored to the current mode, and tail-of-distribution completions are systematically discarded.

Synthetic pre-training augmentation - phi-1 (Gunasekar et al., 2023) used GPT-3.5-generated textbooks to train a 1.3B model with strong HumanEval performance. This works in a single generation because the synthetic data is generated once from a separate, high-quality model and not fed back recursively. The moment you loop this, the quality guarantee dissolves.

The "Model Autophagy Disorder" framing (Alemohammad et al., 2023) characterises the same phenomenon in image generation: without sufficient fresh real data each generation, models either lose precision (quality) or diversity (coverage of the true distribution), or both.

When it falls down

Fresh real data is not a complete fix. Adding even a small fraction of real data to each generation slows collapse but does not eliminate it if the ratio is too low. The required fraction scales with model expressivity: a large model capable of fitting synthetic artefacts more tightly needs a higher real-data proportion to stay stable.

Collapse is asymmetric. High-resource languages, common domains, and majority dialects degrade last. Low-resource languages and speciality domains (medical, legal, niche technical) lose coherent coverage first because they were already in the tail. A model that appears to work fine on benchmark English tasks may have collapsed in its ability to handle code-switching, minority orthographies, or unusual citation formats.

Deduplication does not help. Deduplicating synthetic data removes exact repetition, but does not restore distributional diversity. A model that produces 10,000 paraphrases of the same factual claim is degenerate even if no two outputs are identical.

Collapse is hard to detect mid-training. Perplexity on the synthetic data drops (the model is learning the distribution it is being fed), but perplexity on held-out real data rises. If your evaluation set is also contaminated with synthetic content, you may not notice until deployment.

Theoretical work assumes stationarity. Dohmatob et al. (2024) derive quantitative collapse rates under linear regression, showing progressive degradation with a rate that scales with model capacity divided by real-data sample size. But real training pipelines are non-stationary: they switch datasets, change learning rates, and fine-tune on human feedback. The clean theoretical guarantees do not port directly.

One-shot synthetic augmentation is not recursive training. If a single high-quality generator produces a fixed corpus that is never updated with model outputs, this is not a self-consuming loop and does not exhibit collapse. The confusion between "synthetic data is dangerous" and "recursive synthetic data is dangerous" is the most common misapplication of this literature.

Further reading

• Shumailov, I. et al. (2023). "The Curse of Recursion: Training on Generated Data Makes Models Forget." arXiv:2305.17493. The original model collapse paper; covers VAEs, GMMs, and LLMs with both theory and experiments.
• Alemohammad, S. et al. (2023). "Self-Consuming Generative Models Go MAD." arXiv:2307.01850. Introduces the MAD framing; analyses image generation under three synthetic-data regimes.
• Gerstgrasser, M. et al. (2024). "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data." arXiv:2404.01413. Identifies data accumulation vs. replacement as the key variable; includes theoretical bounds.
• Dohmatob, E., Feng, Y., and Kempe, J. (2024). "Model Collapse Demystified: The Case of Regression." arXiv:2402.07712. Derives quantitative collapse rates via high-dimensional regression; proposes adaptive regularisation as a mitigation.