Web-Scale Corpus Construction

Training GPT-3 consumed roughly 570 GB of filtered text after starting from nearly one trillion tokens of raw crawl data. That ratio - perhaps 40-50% thrown away before a single gradient step - is not waste. It is the job. The quality of a pretraining corpus is arguably the strongest single lever on downstream model capability, yet it receives far less systematic attention than architecture or optimisation.

This concept walks through the five major stages of corpus construction at web scale: extraction, quality filtering, deduplication, mixture design, and decontamination. Tokeniser training is addressed as a downstream consequence of the corpus, not a separate pipeline.

Extraction: from raw crawl to usable text

Common Crawl is the canonical starting point. It archives petabytes of WARC (Web ARChive) files monthly; a single crawl snapshot contains tens of billions of web pages. The extraction step converts HTML into clean, structured plain text.

The dominant tool is trafilatura (and its predecessors jusText, newspaper3k). These libraries strip boilerplate - navigation bars, cookie banners, footer links - and retain the main body text. The challenge is that "main body" is heuristic. A single HTML page might interleave a 200-word article with 1,000 words of sidebar ads; no extractor gets this right universally.

Language identification follows immediately. fastText's language ID model (lid.176.bin) classifies each document into one of 176 languages with high throughput. For English-only corpora, documents falling below roughly 0.65 confidence are discarded. For multilingual corpora the threshold becomes a policy decision per language, because lower-resource languages have noisier crawl coverage.

At this point, the raw corpus is still enormous and largely unusable. Extraction reduces token count by a factor of three to five; the next stages reduce it further.

Quality Filtering

No single definition of "quality" exists. In practice, filtering pipelines combine several families of signal:

Heuristic rules operate on surface statistics. Common examples:

The C4 dataset (used for T5 training) popularised an influential ruleset: keep only documents containing at least three sentences ending in a terminal punctuation mark, discard any document with the phrase "lorem ipsum", and so on. These rules are cheap and surprisingly effective at removing machine-generated spam.

Perplexity filtering scores each document against a small reference language model (typically a KenLM n-gram model trained on Wikipedia or a high-quality seed corpus). Documents with perplexity above a threshold - say, the 30^th percentile of the training set - are discarded. The intuition is that very high perplexity relative to clean text signals garbled language. CCNet introduced this approach; it is standard in most modern pipelines including FineWeb.

Classifier-based filtering trains a binary classifier to distinguish "high quality" text (e.g., Wikipedia, curated books) from general web text. Documents are scored, and a threshold is applied. This is more expensive than heuristics but catches subtler quality signals. The risk is distributional: a classifier trained on Wikipedia as positive examples will penalise any text stylistically unlike Wikipedia, which may exclude legitimate technical writing.

A practical pipeline layers all three: heuristics first (cheapest, removes obvious garbage), then perplexity filtering, then classifier scoring on the surviving subset.

Deduplication

Deduplication is perhaps the most technically interesting stage, and also the one most commonly under-invested. The Lee et al. (2021) finding is striking: over 1% of unprompted model output was verbatim memorised from the training set in models trained on undeduplicated data. Deduplication reduces memorisation, improves perplexity on held-out data, and produces more stable training loss curves.

Three granularities matter:

Exact substring deduplication finds repeated sequences of tokens across documents (suffix arrays over the whole corpus). Computationally expensive: the Lee et al. paper reports processing 100 GB in roughly an hour on 96 CPUs, but multi-trillion-token corpora require careful engineering.

Fuzzy near-duplicate detection uses MinHash LSH (Locality Sensitive Hashing). Each document is represented as a set of n-gram shingles; documents with Jaccard similarity above a threshold (commonly 0.8) are treated as duplicates and all but one copy discarded. MinHash LSH scales to trillions of documents with modest memory.

URL-level deduplication is a cheap first pass: discard documents with identical crawl URLs, or keep only the most recent snapshot of a URL.

The FineWeb paper (Penedo et al., 2024) reports that aggressive MinHash deduplication at the paragraph level, not just document level, yields notable gains in downstream benchmark performance, even when the document-level corpus was already deduplicated.

One subtlety: deduplication should happen before quality filtering in at least one pass, because duplicated boilerplate inflates perplexity statistics and biases classifier training data.

Mixture Design

A corpus is never a single source. The composition of sources - and the relative sampling rates assigned to each during training - is called the data mixture or data recipe. This is where dataset construction becomes closest to an engineering art.

Common sources and their roles:

• Filtered web text (Common Crawl variants): high volume, moderate quality, broad topic coverage.
• Books (Gutenberg, Books3, Books2): long-range coherence, formal prose, narrative structure.
• Code (GitHub, Stack Overflow): structured reasoning, syntax following, tool use.
• Scientific text (arXiv, PubMed, Semantic Scholar): precise language, citation awareness, domain knowledge.
• Wikipedia / encyclopaedias: factual density, consistent structure.

The RefinedWeb paper (Penedo et al., 2023) challenged the assumption that curated sources are strictly necessary: with aggressive filtering and deduplication, web text alone can match or beat mixed corpora. The Pile (Gao et al., 2020) took the opposite position, arguing that 22 diverse sources each provide qualitatively different signal that web data cannot replicate.

In practice, all frontier labs oversample non-web sources relative to their share of available tokens. A document from arXiv might be sampled three or four times per epoch; a web document once. Dolma (Soldaini et al., 2024) and the associated OLMo training transparency work provide some of the most detailed public accounting of these choices.

Mixture weights interact with training compute. The Chinchilla scaling laws describe the compute-optimal frontier for a given model size and token budget, but they were derived on relatively homogeneous data. When the corpus is heterogeneous, the optimal number of training tokens per source is less well characterised.

Decontamination

Evaluation benchmarks (MMLU, HellaSwag, GSM8K, HumanEval, etc.) must not appear verbatim in the training corpus, or benchmark scores become inflated estimates of generalisation. Decontamination is the process of removing benchmark data from the training set.

The standard approach: for each benchmark example, generate a set of n-gram signatures (e.g., 10-gram or 13-gram hashes) and search the entire training corpus for documents containing any matching signature. Matching documents are removed in their entirety (conservative) or the matching passages are removed (surgical).

This sounds simple; in practice it is underestimated. Benchmark questions often appear in blog posts, Reddit threads discussing the question, or model output posted online. A 13-gram match on a canonical answer can flag legitimate educational content. Threshold selection matters: too aggressive and you remove clean training data; too lenient and you leave contamination.

GPT-3 reported contamination analysis as a post-hoc appendix. Most subsequent papers have improved on this, but few have released contamination pipelines. Dolma is an exception: the authors describe an explicit decontamination pass against held-out benchmarks.

Tokeniser Training

The tokeniser is a downstream product of the final corpus. A byte-pair encoding (BPE) or unigram language model tokeniser is trained on a (sampled) subset of the corpus, typically 50 to 200 GB. The vocabulary size is a design choice: 32k (GPT-2), 50k (GPT-¾ extended), 100k+ (Llama 3, Gemma). Larger vocabularies improve compression of high-resource languages and code, but increase embedding table size and softmax cost.

Crucially, the tokeniser should be trained on the filtered and deduplicated corpus, not raw crawl. Training on the raw corpus produces a tokeniser whose vocabulary is polluted by spam tokens and encoding artefacts.

For multilingual corpora, the sampling distribution used for tokeniser training matters more than for monolingual ones. If the training distribution for BPE is 90% English, low-resource languages will be over-segmented (more tokens per word), which effectively gives them less model capacity at fixed context length.

When it falls down

Label leakage at filtering: when a quality classifier is trained on data that partially overlaps with evaluation benchmarks, filtering "improves" quality by inadvertently upweighting benchmark-adjacent text. This is a form of contamination that is invisible to post-hoc decontamination.

Recency bias in crawl data: Common Crawl over-represents recently created or recently linked pages. Pre-2010 web content is systematically underrepresented. This affects model knowledge of historical events, older scientific literature, and cultural content from before the crawl era.

Monolingual monoculture in mixed pipelines: heuristics and classifiers developed on English data degrade in quality for other languages. Perplexity thresholds calibrated on English will discard grammatically correct text in morphologically complex languages (Turkish, Finnish) that naturally produce higher n-gram perplexity.

Deduplication does not equal de-propagation: removing copies of a document from web text does not remove the information it contains if that information is expressed in hundreds of paraphrased versions. Factual claims about named entities can be "deduplicated away" from one phrasing while remaining overrepresented overall.

Mixture instability at scale: the optimal data mixture found by small-scale ablations does not always transfer to the full training run. Source-domain distribution shifts (a source going offline, a crawl snapshot containing a major world event) can silently change the effective mixture mid-training.

Further reading

• Gao et al. (2020), "The Pile: An 800GB Dataset of Diverse Text for Language Modeling" - https://arxiv.org/abs/2101.00027
• Lee et al. (2021), "Deduplicating Training Data Makes Language Models Better" - https://arxiv.org/abs/2107.06499
• Penedo et al. (2024), "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale" - https://arxiv.org/abs/2406.17557
• Soldaini et al. (2024), "Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research" - https://arxiv.org/abs/2402.00159