The Token Budget and the Data Wall

Common Crawl is enormous. The DataComp-LM authors packaged snapshots from 2013 to 2022 into a pool of 240 trillion tokens, and then trained their best model on 2.6 trillion of them (Li et al., 2024, DataComp-LM, arXiv:2406.11794). Roughly ninety-nine percent of the pool was discarded, and the discarding is what made the model good. That gap between what exists and what is usable is the thing people mean when they say the field is running out of data.

Stock, yield and budget are three different numbers

The stock is how much human text exists in a form someone could crawl. Epoch AI puts the indexed web at about 500 trillion words of unique text and expects it to grow by roughly half by 2030 (Epoch AI, 2024, Can AI Scaling Continue Through 2030?).

The yield is what survives extraction, language identification, quality filtering, deduplication and decontamination. RefinedWeb's pipeline removes close to ninety percent of the documents it starts with and still produced five trillion tokens (Penedo et al., 2023, The RefinedWeb Dataset for Falcon LLM, arXiv:2306.01116).

The budget is the yield a particular team can legally, ethically and practically use, in the language mix and the domains it needs. A model aimed at Finnish legal text has a budget measured in gigabytes while the global stock is measured in petabytes.

The data wall is a statement about budgets. The most cited estimate places the effective stock of public human text at around 300 trillion tokens, with a wide ninety percent interval from 100 to 1,000 trillion, and projects that training runs will consume it somewhere between 2026 and 2032 (Villalobos et al., 2024, Will We Run Out of Data?, arXiv:2211.04325).

The budget shrinks while the demand grows

Two trends run against each other. Published token counts per run keep climbing: GPT-3 trained on 300 billion tokens in 2020, Llama 2 on 1.8 trillion, and Llama 3's 405B model on 15.6 trillion (Llama Team, 2024, The Llama 3 Herd of Models, arXiv:2407.21783).

Meanwhile the crawlable commons is closing. An audit of the corpora behind most open models found that in a single year, 2023 into 2024, publishers moved to restrict more than five percent of all tokens in C4 through robots.txt, and more than twenty-eight percent of tokens from the most actively maintained sources in it. Terms of service restrict around forty-five percent of C4 (Longpre et al., 2024, Consent in Crisis, NeurIPS 2024).

A budget that shrinks from one side while demand grows from the other is a planning problem, not a mystery. It has exactly four responses: repeat what you have, filter it less aggressively, substitute other domains, or generate new text. Each has a measured exchange rate, and each is covered in its own concept on this track.

When it breaks

"Running out" is never literal. There is no date on which the last token is consumed. What happens is that the marginal token gets worse, more expensive, or more legally encumbered, so the curve of capability against spend bends. Treat announcements of a cliff with suspicion and ask which budget is binding.

Token counts are not comparable across tokenisers. A corpus is a fixed number of bytes and a variable number of tokens. A vocabulary that is more efficient on your mix reports fewer tokens for the same text, so comparing a 15 trillion token corpus against a 36 trillion token one without naming the tokeniser compares two different measuring sticks.

A bigger budget does not help if it is the wrong data. Adding another trillion tokens of low-grade forum text to a corpus that is already saturated with it changes almost nothing. Budget arguments only work when the tokens being added are both new and in a distribution the model needs.

Multimodal stock is not a drop-in substitute. Video and audio add enormous nominal token counts, and the Epoch analysis treats them as a plausible multiplier on the total, but text capability does not improve in proportion to frames ingested. Counting all modalities in one number flatters the supply side.