The Data Wall Is a Budget: Pretraining When Unique Tokens Run Short

Two hundred and forty trillion tokens. That is the size of DCLM-Pool, the standardised Common Crawl extract the DataComp-LM authors released so that dataset designs could be compared on equal footing. The training set that won their own benchmark, DCLM-Baseline, holds about 2.6 trillion tokens. Roughly ninety-nine percent of the pool was discarded, and the discarding is the reason the resulting 7B model reached around 63 to 64 percent on 5-shot MMLU (Li et al., 2024, DataComp-LM, arXiv:2406.11794).

Hold that next to the standard framing of the data wall, which says the field is about to run out of text. Both are true at once. Raw text is abundant; usable, unique, in-distribution text is scarce, and the gap between them is a pipeline with a yield near one percent. What gets called a wall is a budget constraint with four known levers and measured exchange rates between them.

Why this matters: Every pretraining decision downstream of corpus construction inherits the token budget as a hard parameter. If you treat unique data as inventory, you under-spend compute, over-filter the corpus, or discover at 4 trillion tokens that you are on your tenth pass over a shard that fits in a laptop. If you treat it as a budget, repetition, filter strength, domain mix and synthetic rewriting become one allocation problem with published numbers attached to each option.

TL;DR

• The practical yield of a web pipeline is about one percent of the raw pool: DCLM-Pool's 240T became a 2.6T training set, and FineWeb's 15T became 1.3T after an educational-quality filter (Penedo et al., 2024, arXiv:2406.17557).
• Repeating a corpus for up to four epochs changes loss negligibly versus the same count of unique tokens, and past that the value of extra compute decays towards zero (Muennighoff et al., 2023, arXiv:2305.16264).
• Repeating part of a corpus is a different operation entirely. An 800M parameter model was degraded to the quality of a 400M one by repeating 0.1% of its data 100 times (Hernandez et al., 2022, arXiv:2205.10487).
• Filter strength is a joint decision with the compute budget, not a data-quality decision: hard filtering dominates when data is plentiful, relaxing filters wins when data binds.
• Code is a measured substitute: up to half the training mix, about 42B tokens of Python in the study that tested it, with no natural-language degradation. Past that share, language performance drops quickly.
• Synthetic rewriting adds density, not facts. Nemotron-CC ships 4.4T real plus 1.9T synthetic tokens and reports an 8B model beating Llama 3.1 8B by 5 points on MMLU (Su et al., 2024, arXiv:2412.02595).
• Model collapse follows from replacing real data with generated data, not from using generated data: accumulating synthetic beside real bounds the error instead of compounding it.
• The commons is closing faster than the stock grows: in one year publishers restricted over 5% of all C4 tokens, and over 28% of tokens from its most actively maintained sources, via robots.txt (Longpre et al., 2024).

At a Glance

flowchart LR
    A["Raw crawl<br/>240T tokens"] --> B["Extract, filter<br/>dedup, decontaminate"]
    B --> C["Unique budget U<br/>about 1 percent"]
    C --> D["Repeat<br/>E epochs"]
    C --> E["Relax filters<br/>raise U"]
    C --> F["Substitute<br/>code, languages"]
    C --> G["Synthesise<br/>rephrase U"]
    D --> H["Effective tokens D-prime"]
    E --> H
    F --> H
    G --> H
    H --> I["Training run<br/>D tokens of compute"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class A blue
    class B purple
    class C amber
    class D,E,F,G teal
    class H purple
    class I slate

The diagram holds the whole argument. Only the raw crawl is fixed by nature. The amber node is what every other decision is denominated in, and the four teal boxes are the only moves available once it is known.

Before the Token Budget Mattered

For the first few years of large model training nobody allocated a token budget, because compute was the binding constraint and the corpus was larger than any run could consume. The only question about data was how aggressively to clean it.

GPT-3's data table is the first public artefact of a team allocating repetition by hand. Forty-one monthly Common Crawl shards, 45 terabytes of compressed plaintext, became 570 gigabytes after filtering, about 400 billion byte-pair-encoded tokens. The run then used 300 billion tokens from a weighted mix in which filtered Common Crawl was sampled 0.44 times and English Wikipedia 3.4 times (Brown et al., 2020, arXiv:2005.14165). In 2020, three and a half epochs over the best shard was obviously correct, and nobody called it repetition.

Chinchilla changed the question. Training over 400 models from 70 million to 16 billion parameters on 5 to 500 billion tokens, Hoffmann and colleagues showed that model size and training tokens should scale together, then demonstrated it with a 70 billion parameter model on 1.4 trillion tokens that beat the 280 billion parameter Gopher at equal compute (Hoffmann et al., 2022, arXiv:2203.15556). Roughly twenty tokens per parameter, and data became a first-class constraint rather than a preprocessing chore (How Much Data Is Enough? The Chinchilla Correction). Once the recipe asks for tokens in proportion to parameters, somebody has to count them.

timeline
    title From Abundance to Allocation
    2020 : GPT-3 trains on 300B tokens from a hand-weighted mix
         : Wikipedia at 3.4 epochs; filtered Common Crawl at 0.44
    2022 : Chinchilla shows tokens and parameters should scale together
         : Hernandez et al. show a repeated 0.1 percent can halve effective model quality
         : Villalobos et al. first project exhaustion of public human text
    2023 : RefinedWeb removes nearly 90 percent of documents and still yields 5T tokens
         : Muennighoff et al. measure the value of a repeated token; four epochs is near-free
    2024 : FineWeb publishes 15T tokens and a 1.3T educational subset
         : DataComp-LM shows model-based filtering dominates dataset design
         : Nature publishes the model collapse result; Consent in Crisis measures the closing commons
    2025 : Nemotron-CC ships 1.9T synthetic tokens beside 4.4T real ones
         : Qwen3 reports 119 languages; model cards describe a corpus near 36T tokens

[IMAGE: Log-scale line chart, 2020 to 2025, of published pretraining token counts per flagship run (GPT-3 300B, Chinchilla 1.4T, Llama 2 1.8T, Llama 3 405B 15.6T, Qwen3 circa 36T) against a flat band for the estimated 100T to 1000T stock of public human text. Caption: "Demand per run grew two orders of magnitude; the stock did not."]

What a Token Budget Is Made Of

A token budget is the product of a retention cascade and the input to an allocation problem. Both halves have arithmetic worth doing by hand.

The yield cascade

Every published pipeline has the same shape: extract text from HTML, identify the language, apply heuristic filters, apply a learned quality filter, deduplicate, decontaminate. Each stage multiplies a retention rate below one, so the output is a product, which is why small per-stage differences produce order-of-magnitude differences in yield.

RefinedWeb reports its stages in the open: about half of documents dropped for not being English, a quarter of the remainder for insufficient quality, twelve percent of what survives as duplicates, with the full pipeline removing nearly ninety percent of the documents it started with (Penedo et al., 2023, arXiv:2306.01116). FineWeb then ran an educational-quality classifier over its own 15 trillion token output and kept 1.3 trillion, under nine percent.

Six stages at sixty percent retention each leave about five percent. That is the arithmetic that turns an abundant web into a scarce corpus, and it also shows where the gains are: lifting one stage from sixty to eighty percent retention raises total yield by a third.

What a repeated token is worth

With \(U\) unique tokens in hand and a compute budget calling for \(D > U\) training tokens, the practical question is what the second pass is worth. Muennighoff and colleagues answered it with roughly 400 runs up to 9 billion parameters and 900 billion tokens: up to four epochs of repeated data yields negligible loss change versus that many unique tokens, and beyond that the value of extra compute decays towards zero (Muennighoff et al., 2023, arXiv:2305.16264).

The model behind the result treats each pass as worth a constant fraction less than the one before. With per-repetition decay \(\delta\),

where \(D'\) is effective unique data, \(E\) is the epoch count, and the paper summarises the decay as \(R_D^{*} = (1-\delta)/\delta\), the scale beyond which repeated tokens stop contributing. Substituting \(D'\) for \(D\) in the Chinchilla parametric loss,

turns compute-optimal scaling into something usable when data runs out first. Here \(N\) is parameters, \(L_{\infty}\) the irreducible loss, and \(A\), \(B\), \(\alpha\), \(\beta\) fitted constants. Under the data-constrained fit, extra compute is best spent raising both the epoch count and the parameter count rather than either alone.

Two conclusions follow from the shape of the sum rather than any fitted constant, which is what makes them worth trusting. Early epochs are nearly free, since \((1-\delta)^{k-1}\) stays close to one while \(k\) is small. And the sum is bounded above by \(U/\delta\), a hard ceiling on what repetition can buy at any compute budget.

Why uniform repetition is safe and selective repetition is not

The four-epoch result is about repeating a whole corpus. Repeating a slice of it behaves nothing like the average suggests.

Anthropic's study duplicated a small fraction of the corpus many times and found that repeating 0.1% of the data 100 times degraded an 800 million parameter model to the quality of a model half its size, the rest of the tokens unique (Hernandez et al., 2022, arXiv:2205.10487). They also report a pronounced double descent, test loss rising part-way through training before recovering, with interpretability evidence that the damage lands on copying and induction structures. Their reading is that an intermediate repetition range is the most harmful: memorising a repeated shard consumes capacity that would otherwise go to generalising circuits.

The engineering consequence is sharp. Two pipelines reporting the same average epoch count can be in completely different regimes, because what matters is the distribution of per-document repeat counts rather than its mean. Local-only deduplication is how teams acquire a bad distribution without choosing one: pages recurring across monthly snapshots become a high-multiplicity shard nobody planned.

Filtering is an allocation decision

The literature appears to contradict itself on filter strength, and the contradiction is informative.

When unique data is plentiful relative to compute, filtering hard wins decisively, which is DataComp-LM's central result: model-based filtering is the dominant lever in dataset design, and 2.6 trillion selected tokens beat much larger permissive sets. When data is scarce relative to compute the sign flips, and the data-constrained study found that dropping common filters, including perplexity filtering and some deduplication, can be the better move. Tokens a strict filter discards are tokens you will otherwise manufacture by repetition, at the decay rate above.

Both results are correct because filter strength is not a property of the data. What is being optimised is the quality of the effective corpus the model sees, repetitions included, and that depends on the compute budget. A threshold chosen without reference to the number of training tokens was chosen for someone else's run.

[IMAGE: Two-panel line chart. Left panel: effective tokens D-prime against epoch count E for three values of delta (0.05, 0.1, 0.2), showing each curve bending towards its U/delta ceiling. Right panel: the same curves expressed as efficiency, D-prime divided by E times U, showing all three near 1.0 for E below 4 and falling steeply after. Caption: "The first four passes are nearly free in every parameterisation; the ceiling is set by delta alone."]

Seeing It in Motion

Two questions are best answered in diagram form: which lever to pull, and where in a pipeline run the decision gets made.

flowchart TB
    S["Compute budget needs D tokens<br/>pipeline yields U unique"] --> R{"D divided by U"}
    R -->|"under 4"| A["Repeat and stop<br/>nearly free"]
    R -->|"4 to 10"| B{"Filter threshold<br/>already permissive?"}
    R -->|"over 10"| C["Repetition alone wastes compute"]
    B -->|no| D["Relax filter<br/>raise U, re-ablate"]
    B -->|yes| E["Substitute domains<br/>code, more languages"]
    C --> F["Synthesise from U<br/>pay inference cost"]
    C --> E
    D --> G["Re-measure U, recompute epochs"]
    E --> G
    F --> G

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff

    class S blue
    class R,B purple
    class A emerald
    class C amber
    class D,E,F teal
    class G purple

The ratio \(D/U\) is the only diagnostic needed to know which conversation to have. Below four the budget is not binding and the right move is to stop optimising it. Above ten no amount of repetition rescues the run, and the remaining options cost either quality or inference compute. The ablation loop below is what separates a measured corpus from an inherited one.

sequenceDiagram
    participant C as Crawl store
    participant P as Pipeline
    participant Q as Quality filter
    participant A as Ablation trainer
    participant M as Mixer
    C->>P: 96 snapshots of raw HTML
    P->>P: extract, language ID, heuristics, dedup
    P->>Q: candidate corpus
    Q->>A: corpus at threshold t
    A->>A: short run matched on training tokens
    A-->>Q: benchmark delta per threshold
    Note over Q,A: repeat for three to five thresholds<br/>equal tokens, not equal corpus size
    Q->>M: chosen threshold, unique budget U
    M->>M: set epochs, code share, language mix
    M-->>C: request re-extraction if U is short

[IMAGE: Mock ablation dashboard, five filter-threshold rows, columns for unique tokens retained, implied epochs at a fixed 1T budget, effective tokens, and MMLU delta. The best effective-token row and the best MMLU row differ, highlighted in contrasting colours. Caption: "The threshold that maximises effective tokens is not always the one that maximises the metric you ship."]

[IMAGE: Sankey diagram of a six-stage web pipeline, left edge 100 percent of raw documents, with labelled flows peeling off at extraction, language identification, heuristic filters, classifier filtering, deduplication and decontamination, ending in a narrow band marked "unique budget U". Each reject flow carries its own percentage. Caption: "Retention rates multiply; the exit band is the only number the training run sees."]

Watch It Run

By the Numbers

Sources: DCLM (Li et al., 2024); FineWeb and FineWeb-Edu (Penedo et al., 2024); RefinedWeb (Penedo et al., 2023); Nemotron-CC (Su et al., 2024); GPT-3 (Brown et al., 2020); Chinchilla (Hoffmann et al., 2022); Llama 2 and Llama 3 counts as reported in the Llama 3 paper (Llama Team, 2024). Tokens-per-parameter figures are computed, not quoted, and corpus token counts use different tokenisers, so they are not strictly comparable. The 300 trillion token stock estimate, interval 100 to 1,000 trillion, is explicitly an estimate (Villalobos et al., 2024).

A Concrete Example

A team has compute for a 7 billion parameter model at 4 trillion training tokens, which is about \(6ND \approx 1.7 \times 10^{23}\) FLOPs. Their pipeline, tuned to keep the top ten percent by classifier score in their target language, yields 400 billion unique tokens. The ratio \(D/U\) is 10, so repetition alone will not be free.

Take the geometric-decay model with \(\delta = 0.1\), an illustrative value rather than a fitted constant from the paper, chosen so the arithmetic can be replayed by hand. The pattern it produces matches the published result that four epochs are near-free.

Step 1: price the repetition. Effective tokens after \(E\) epochs are \(D' = 400 \cdot (1 - 0.9^{E}) / 0.1\) billion.

At the ten epochs their budget implies, 4 trillion tokens of compute buys the effect of 2.6 trillion unique ones: thirty-five percent of the run pays for passes worth less than the first.

Step 2: price the filter. They re-run the classifier keeping the top twenty-five percent instead of the top ten, lifting \(U\) to 1 trillion. Now \(E = 4\) and \(D' = 1000 \cdot 3.439 = 3.44\) trillion effective tokens, a 32 percent gain for zero additional compute. The cost is average document quality, so this is a claim to test rather than assume: two ablations at 150 billion training tokens each, matched on tokens and not corpus size, settle it in a day of cluster time.

Step 3: price the substitution. Instead of relaxing the filter they add 400 billion tokens of permissively licensed code, taking the mix to half code, which is the share the data-constrained study tested without natural-language degradation. Now \(U = 800\)B, \(E = 5\), and \(D' = 800 \cdot 4.095 = 3.28\) trillion. Nearly the same gain as the filter relaxation, with a different risk: the model becomes measurably better at code and the natural-language headroom beyond a fifty percent share is gone.

Step 4: price the synthesis. They rephrase their 400 billion high-quality tokens into 600 billion rewritten ones for \(U = 1\) trillion nominal. The rewrites are not independent of their sources, so the effective gain is smaller than the count suggests, and generating them is an inference run costing a few percent of the training FLOPs. This is the lever to reach for once the others are spent, which is how Nemotron-CC uses it: 1.9 trillion synthetic tokens beside 4.4 trillion real ones.

The decision. Steps 2 and 3 are nearly free and go first, in that order, since the filter change costs only ablation time. Step 1 alone wastes a third of the compute. Step 4 works and is not free. A fifth option, spending the same compute on a smaller model at fewer epochs, deserves a line in the same table rather than a separate argument.

[IMAGE: Waterfall chart from 400B unique tokens showing each lever's contribution to effective tokens: repetition to 10 epochs (+2.21T, partly shaded as waste), filter relaxation (+830B), code substitution (+670B), synthetic rewriting (a discounted bar marked with a question mark). Caption: "Additive in effect, very different in cost."]

Where It Breaks

The repeat distribution, not the repeat average

A uniform four-epoch schedule and a four-epoch average containing a hundredfold-repeated shard are different training regimes, and nothing in a token count distinguishes them. The defence is to compute the histogram of per-document multiplicity after deduplication and before training, then treat its tail as a bug rather than a statistic.

Memorisation and contamination scale with epochs

Verbatim extraction risk rises with duplication count, so an epoch budget is simultaneously a privacy and copyright decision: deduplication reduces that exposure and repetition puts it back, proportionally. Any evaluation text that survived decontamination behaves the same way, with each pass memorising it more thoroughly and the benchmark improving for a reason that will not transfer.

Synthetic data collapses under replacement

The Nature result is unambiguous about the regime it tested: train generation after generation on the previous generation's output and the distribution's tails disappear irreversibly, in language models, variational autoencoders and Gaussian mixtures alike (Shumailov et al., 2024, Nature 631, 755-759). The follow-up work localises the cause to replacement: when each generation's synthetic data accumulates alongside the original real data, test error has a finite upper bound independent of the iteration count (Gerstgrasser et al., 2024, arXiv:2404.01413). Keeping the real corpus in the mix is the whole difference between bounded and unbounded error.

The budget shrinks on its own

Corpus planning tends to assume a static stock. An audit of the data behind most open models found that in one year, 2023 into 2024, publishers fully restricted over five percent of all C4 tokens and over twenty-eight percent of tokens from its most actively maintained sources via robots.txt, with terms of service covering around forty-five percent of C4, and forecast a further 2 to 4 percent of corpus tokens within the following year (Longpre et al., 2024). A snapshot is a depreciating asset for legal reasons, independent of its technical quality.

Effective-token accounting invites self-deception

Every term in \(D' = U(1-(1-\delta)^E)/\delta\) is estimated. \(U\) depends on the tokeniser, \(\delta\) is fitted on someone else's models and data, and \(D\) assumes a utilisation the cluster may not deliver. Treat the output as a ranking of options, not a prediction of loss. Reports from other settings suggest architectures differ in how many passes they tolerate, with sparse models said to tolerate fewer than dense ones of comparable capacity, which has not been independently established at scale.

Short ablations favour corpora that saturate late

Ablating filter thresholds at a few hundred billion tokens compares corpora that are all still on their first pass, and the phenomenon under study only appears after the data has been seen once. An ablation that never reaches the real run's epoch count cannot measure the effect it exists to measure, which makes the ablation horizon a first-class parameter rather than a budget detail.

[IMAGE: Histogram of per-document repeat multiplicity for two corpora with identical four-epoch averages. The first is a single spike at 4. The second is a tall spike at 1 with a long thin tail reaching past 100, with the tail region shaded rose. Caption: "Two corpora, one average, two very different training regimes."]

Alternative Designs

The last two rows get left out of data-wall conversations and are often the right answer. If a run needs 4 trillion tokens because a 7 billion parameter model has to know a great deal, an index plus a smaller model may ship the same product for less, and the budget stops binding because the requirement changed.

How It Is Used in Practice

Open corpus releases now read as budget documents. FineWeb publishes both the 15 trillion token corpus and the 1.3 trillion token educational subset because the right choice depends on the consumer's compute budget, and ships its ablation models so the choice can be re-made rather than inherited.

Nemotron-CC is the clearest production use of the substitution lever: 4.4 trillion globally deduplicated real tokens plus 1.9 trillion synthetically derived ones from 99 snapshots. NVIDIA reports an 8 billion parameter model trained for 15 trillion tokens, 7.2 trillion of them from this dataset, beating Llama 3.1 8B by 5 points on MMLU and 3.1 on ARC-Challenge, and claims four times more unique real tokens than DCLM at matched MMLU. That is the data-wall argument stated as a product feature.

At the frontier, disclosure stops where it gets interesting. Llama 3's 405B model is documented at 15.6 trillion training tokens and roughly \(3.8 \times 10^{25}\) FLOPs, about 39 tokens per parameter, well past Chinchilla's ratio and in the overtrained regime where inference economics set the target. Qwen3 reports expanding from 29 to 119 languages, its roughly 36 trillion token corpus figure appearing on the released model cards rather than in the technical report (Qwen Team, 2025, arXiv:2505.09388). Nobody publishes a unique-token count after deduplication, the one number that would settle how close the wall is.

Vendor claims here deserve the usual discount. DatologyAI reports its BeyondWeb synthetic corpus training up to 7.7 times faster than open web data, and a 3 billion parameter model on 180 billion of its tokens beating an 8 billion parameter model on the same budget of Cosmopedia (DatologyAI, 2025, arXiv:2508.10975). Self-reported, against other synthetic sets, by a company selling data curation, and awaiting replication. The direction matches Nemotron-CC; the multiple is not a planning constant.

[IMAGE: Horizontal bars comparing published corpora by composition, each segmented into real deduplicated, synthetic and non-English tokens where disclosed, with an "undisclosed" segment for frontier runs. Caption: "Corpus composition is now a competitive disclosure decision."]

Insights Worth Remembering

1. Unique data is a budget, not an inventory. An inventory is drawn down and then empty; a budget is allocated across competing uses at known exchange rates. Nearly every published result in this area is an exchange rate between four levers: repetition, filter strength, domain mix and synthesis.

2. The yield, not the stock, is the scarce quantity. A 240 trillion token pool yielding a 2.6 trillion token training set says nothing about the web being too small. It describes a cascade of multiplicative retention rates, which is an engineering target rather than a natural limit.

3. Four epochs is a licence for uniform repetition only. One aggregate epoch count can describe a nearly free regime or one that halves effective model quality. What matters is the distribution of per-document repeat counts, and it is absent from every published token total.

4. A filter threshold inherited from a data-abundant regime is wrong in a constrained one. The threshold is a joint decision with compute, and teams routinely import one from a paper whose compute-to-data ratio was nothing like theirs, then pay for it in wasted epochs.

5. Synthetic data is a presentation improvement, not a knowledge source. Rephrasing makes existing information denser and better formatted, which measurably speeds training. It adds no facts, and if it replaces real data rather than accumulating beside it, it destroys the distribution's tails.

6. Deduplication quality determines repetition quality. Local-only deduplication is how teams end up with a high-multiplicity shard they never chose. A global MinHash index is better understood as the price of controlling the repeat distribution than as a storage optimisation.

7. The corpus is a depreciating asset for legal reasons. Robots.txt and terms-of-service restrictions closed off a measurable share of the most valuable sources within one year. A corpus built today cannot necessarily be rebuilt tomorrow, which makes archival and provenance tracking strategic rather than a compliance chore.

Open Questions

Does the decay constant transfer across architectures? It was measured for dense decoder-only models on one data mix. Sparse models are reported to tolerate fewer passes, which if it holds means mixture-of-experts architectures reach the wall sooner than their active parameter count suggests. Nothing like the original 400-run sweep has tested it, so this is an open empirical question rather than a known fact.

How much does synthetic rewriting add, net of its sources? The measured speedups are real and the mechanism, denser presentation of existing information, is understood. Nobody has published a clean decomposition separating the gain from reformatting, from curriculum effects, and from the generator's own knowledge leaking into the rewrites. Without it, synthesis is the least trustworthy of the four exchange rates.

Is there a filter objective that is not benchmark-shaped? Model-based filters score documents by resemblance to curated reference sets, with thresholds tuned on MMLU-style gains, which selects for encyclopaedic prose. Whether a filter can be trained against something the field cares about more directly, such as downstream reasoning or long-context behaviour, is open, and its absence is a plausible reason corpora feel increasingly similar.

What is the real frontier unique-token count? Published figures are training tokens, not unique tokens, and no frontier lab discloses the latter. The difference between a 36 trillion token run at one epoch and the same run at three epochs over 12 trillion unique tokens is the whole question of how close the wall is, and it is unanswerable from public information.

Does multimodal data help text capability, or only token totals? Epoch AI treats image, video and audio as a plausible multiplier on available training data, which is correct as accounting. Whether ingesting video improves the textual reasoning models are judged on is a separate claim, and the evidence is thin relative to how often the multiplier is cited.

Sources and Further Reading

1. Brown, T., et al. (2020). "Language Models are Few-Shot Learners." NeurIPS 2020. arXiv:2005.14165
2. Hoffmann, J., et al. (2022). "Training Compute-Optimal Large Language Models." arXiv:2203.15556
3. Hernandez, D., et al. (2022). "Scaling Laws and Interpretability of Learning from Repeated Data." arXiv:2205.10487
4. Muennighoff, N., Rush, A. M., et al. (2023). "Scaling Data-Constrained Language Models." NeurIPS 2023; also JMLR 26 (2025). arXiv:2305.16264
5. Penedo, G., et al. (2023). "The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data Only." NeurIPS 2023. arXiv:2306.01116
6. Penedo, G., et al. (2024). "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale." NeurIPS 2024 Datasets and Benchmarks. arXiv:2406.17557
7. Li, J., Fang, A., et al. (2024). "DataComp-LM: In Search of the Next Generation of Training Sets for Language Models." NeurIPS 2024. arXiv:2406.11794
8. Su, D., et al. (2024). "Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset." ACL 2025. arXiv:2412.02595
9. Villalobos, P., et al. (2024). "Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data." ICML 2024. arXiv:2211.04325
10. Longpre, S., Mahari, R., Lee, A., Lund, C., et al. (2024). "Consent in Crisis: The Rapid Decline of the AI Data Commons." NeurIPS 2024. arXiv:2407.14933
11. Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024). "AI Models Collapse When Trained on Recursively Generated Data." Nature 631, 755-759. doi:10.1038/s41586-024-07566-y
12. Gerstgrasser, M., et al. (2024). "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data." arXiv:2404.01413
13. Llama Team, Meta AI (2024). "The Llama 3 Herd of Models." arXiv:2407.21783
14. Qwen Team (2025). "Qwen3 Technical Report." arXiv:2505.09388
15. DatologyAI (2025). "BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-Scale Pretraining." arXiv:2508.10975
16. Epoch AI (2024). "Can AI Scaling Continue Through 2030?" epoch.ai

Related reading: How Much Data Is Enough? The Chinchilla Correction, 250 Documents: Why Data Poisoning Gets Easier as Models Get Bigger, and the pretraining-data concepts on the token budget, corpus yield, repetition, filter strength and domain substitution.

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.