Domain Substitution When Web Text Runs Short

Once a corpus has been filtered and the epoch budget is set, one lever remains: change what is in the mix. Three substitutions have measured exchange rates, and all three are already standard in published recipes rather than speculative.

Code as filler that pays for itself

The data-constrained scaling study tested filling the gap with Python from The Stack and found that code could occupy up to half the training mix, around 42 billion tokens in their setting, with no deterioration on natural-language benchmarks. Past that share, natural-language performance falls quickly (Muennighoff et al., 2023, arXiv:2305.16264).

Read as an exchange rate, this is close to a doubling of the usable budget for teams whose shortage is in natural-language tokens. The caveat the authors state explicitly is that their benchmark suite did not cover non-natural-language tasks, so the result bounds the harm rather than measuring the benefit.

Multilingual text as a genuine expansion

Language identification is usually the largest single cut in a pipeline, which means a monolingual corpus is a small slice of its own crawl. Reversing that decision converts discarded tokens into training data: Qwen3 reports expanded support from 29 to 119 languages and dialects, with model cards describing a corpus of roughly 36 trillion tokens (Qwen Team, 2025, Qwen3 Technical Report, arXiv:2505.09388; the token figure appears on the released model cards rather than in the paper abstract).

The cost is interference. Fixed capacity divided across more languages can reduce per-language quality, and the balancing problem that follows is a design task in its own right rather than a free addition.

Generated text as manufactured tokens

The third substitution rewrites what you already have. Nemotron-CC combines 4.4 trillion globally deduplicated real tokens with 1.9 trillion synthetically derived ones from 99 Common Crawl snapshots, and reports an 8 billion parameter model trained on a 15 trillion token budget beating Llama 3.1 8B by 5 points on MMLU (Su et al., 2024, Nemotron-CC, arXiv:2412.02595). DatologyAI's BeyondWeb reports up to 7.7 times faster training than open web data from targeted rephrasing, a self-reported figure awaiting independent replication (DatologyAI, 2025, BeyondWeb, arXiv:2508.10975).

The mechanism is worth stating plainly: rephrasing does not add information, it re-presents existing information in a denser, better-formatted distribution. That is genuinely useful, and it is not a source of new facts.

When it breaks

Replacement collapses, accumulation does not. Training generation after generation purely on the previous generation's output destroys the tails of the distribution, which is the model collapse result (Shumailov et al., 2024, Nature 631, 755-759). The follow-up finding is that the collapse depends on replacement: accumulating synthetic data alongside the original real data bounds the error instead of letting it grow with each iteration (Gerstgrasser et al., 2024, arXiv:2404.01413). Keep the real corpus in the mix and the risk is manageable; discard it and it is not.

Generation costs inference compute. Rewriting a trillion tokens is a large inference run with its own bill, so synthetic tokens are not free in the way a relaxed filter threshold is. The comparison to make is cost per effective token against the alternative of another epoch.

Self-reported speedups measure a package, not a technique. Vendor comparisons bundle generation prompts, source selection, filtering and mixing. Without ablations isolating each, the headline multiple does not transfer to a different pipeline.

Substitution changes what the model is. A mix that is half code, heavily multilingual, or substantially rewritten produces a different model, not a cheaper version of the same one. Evaluate the substitution on the capabilities you ship, not only on the loss curve that motivated it.