Corpus Yield From Crawl to Training Tokens

Every published web corpus is the output of the same shape of pipeline, and every stage of it multiplies a retention rate below one. The headline number a team cares about, unique trainable tokens, is the product of six or seven such rates, which is why two pipelines over the same crawl can differ in output by an order of magnitude without either being wrong.

The cascade

GPT-3's authors reported the arithmetic in the clearest form available: 45 terabytes of compressed plaintext from 41 monthly Common Crawl shards became 570 gigabytes after filtering, about 400 billion byte-pair-encoded tokens (Brown et al., 2020, Language Models are Few-Shot Learners, arXiv:2005.14165). Roughly one part in eighty survived, by bytes.

Later pipelines are more explicit about where the loss happens. RefinedWeb discards about half its documents for not being English, a further quarter of the remainder for insufficient quality, and twelve percent of what is left as duplicates, and reports that the full pipeline removes nearly ninety percent of the documents it started with (Penedo et al., 2023, arXiv:2306.01116). FineWeb ran 96 Common Crawl snapshots through text extraction, filtering and deduplication to produce 15 trillion tokens, then applied an educational-quality classifier to produce FineWeb-Edu at 1.3 trillion, under nine percent of its own input (Penedo et al., 2024, The FineWeb Datasets, arXiv:2406.17557).

The compounding is the point. Six stages each retaining sixty percent leave five percent. A team that doubles the retention of one stage from sixty to eighty percent raises the whole yield by a third, which is a large return for a bounded engineering effort.

Which stage dominates

The stages are not equal, and which one dominates depends on the target.

Text extraction sets the ceiling. Boilerplate removal that is too aggressive truncates articles; too permissive and the corpus fills with navigation chrome. The loss here is invisible downstream because what was dropped never appears in a count.

Language identification is the largest single cut for a monolingual corpus and the most consequential for a multilingual one. Classifier thresholds trade recall on low-resource languages against precision on the dominant one.

Deduplication removes the most tokens in absolute terms on a multi-snapshot crawl, since the same pages recur month after month. Whether it is applied globally across snapshots or locally within each one changes the output substantially: Nemotron-CC reports 4.4 trillion globally deduplicated real tokens from 99 snapshots (Su et al., 2024, Nemotron-CC, arXiv:2412.02595).

Quality filtering is the stage with the widest defensible range, from heuristics that drop a few percent to classifier thresholds that keep under a tenth.

When it breaks

Yield is reported in incompatible units. Papers variously report documents, bytes, and tokens, before or after deduplication, with different tokenisers. A retention rate quoted in documents can be very different from the same stage's retention in tokens, because filters correlate with length.

Global deduplication is expensive and it hides a choice. Exact and fuzzy dedup across snapshots needs a MinHash index over hundreds of billions of documents. Teams that cannot afford it deduplicate within snapshots and silently keep cross-snapshot duplicates, which then act as repeated data in training.

The stages interact. Running the quality classifier before deduplication gives a different corpus than running it after, because duplicate-heavy spam shifts the classifier's score distribution. Pipeline order is a design decision that is rarely ablated and frequently inherited.

Decontamination is the stage most often skipped. Removing evaluation set text is cheap compared with the other stages and it is the only one whose omission makes results look better rather than worse, which is exactly why it needs to be checked rather than assumed.