Data Pipelines at Scale

Roughly 99% of the raw bytes scraped from a CommonCrawl snapshot never make it into a model's training set. The Falcon team extracted five trillion tokens from CommonCrawl but publicly released only 600 billion after filtering (Penedo et al., 2023). That 8-to-1 discard ratio is not waste; it is the core engineering problem of pretraining data.

From the Crawl to Clean Text

Every large-scale pretraining pipeline starts from the same commodity source: Common Crawl's petabyte-scale WARC archives, updated monthly since 2008. The raw bytes are HTML-wrapped boilerplate, encoded in at least a dozen character sets, and riddled with duplicate near-duplicates of the same article published across thousands of mirror sites.

The extraction stage transforms WARCs into plain Unicode text. trafilatura, resiliparse, and custom rule-based extractors strip nav bars, cookie banners, and boilerplate markup. Language identification (typically fastText LangDetect) follows immediately, because later quality heuristics are language-specific.

Quality filtering then applies a cascade of cheap signal checks before any expensive model inference:

Perplexity filtering with a small n-gram LM (a KenLM 5-gram trained on Wikipedia) is especially powerful: a document whose perplexity far exceeds the distribution learned from clean prose almost certainly contains garbled OCR, machine-translated spam, or auto-generated affiliate content.

Deduplication: Why One Copy Is Enough

Identical or near-identical content at scale causes two separate harms. First, a model repeatedly seeing the same passage memorises it verbatim; Lee et al. (2022) showed that models trained on deduplicated C4 emitted memorised text ten times less frequently and needed fewer steps to reach the same perplexity. Second, test-set contamination (covered below) becomes harder to audit when the corpus is full of near-duplicates that differ only in whitespace.

Two algorithms dominate at scale:

MinHash LSH (Locality-Sensitive Hashing). Each document is represented as a set of n-gram shingles; MinHash sketches approximate Jaccard similarity. Documents exceeding a similarity threshold (commonly 0.8) are collapsed to a single representative. MinHash scales near-linearly and works well for paragraph-level fuzzy duplicates.

Suffix-array exact deduplication. The entire corpus is concatenated and sorted as a suffix array. Shared substrings of length above some threshold (e.g., 100 tokens) are identified and the duplicate spans removed from all but one document. This catches exact repeated passages that differ in surrounding context, which MinHash misses.

Both are applied in the RefinedWeb pipeline. The choice of threshold matters: an overly aggressive Jaccard cutoff can delete legitimately similar news summaries; too loose, and you leave near-duplicates intact.

Data Mixture: The Token Budget Problem

After Hoffmann et al. (2022, "Chinchilla") established that compute-optimal training requires roughly one training token per parameter, practitioners quickly discovered that web text alone does not fill that budget with the right distribution. Web crawls are 80-90% English prose of variable quality; models trained on them are noticeably weaker on code, maths, and multilingual understanding.

A mixture weights different source domains:

total_tokens = sum_i(w_i * corpus_i_tokens)

where sum_i(w_i) = 1.0
and each w_i is tuned on held-out perplexity / downstream eval

LLaMA (Touvron et al., 2023) used the following approximate mixture: CommonCrawl 67%, C4 15%, GitHub 4.5%, Wikipedia 4.5%, books 4.5%, ArXiv 2.5%, StackExchange 2%. The Pile (Gao et al., 2021) formalised this earlier with 22 sub-corpora weighted by domain diversity rather than raw size.

Choosing mixture weights is an active research problem. Practitioners run small ablations at ~1B tokens to estimate downstream impact before committing to a trillion-token run.

Tokeniser Training and Its Blind Spots

The tokeniser is trained on a sample of the final corpus (typically 50-100 GB), not on raw crawl data. This distinction matters: if you train the tokeniser on the unfiltered crawl, rare-language scripts and code identifiers inflate the vocabulary with high-frequency boilerplate tokens.

BPE (Byte-Pair Encoding) is the dominant algorithm. Training proceeds by iteratively merging the most frequent byte-pair until the vocabulary reaches the target size (32k to 128k tokens). Sentencepiece wraps BPE with Unicode normalisation and handles whitespace as a first-class character.

Key design decisions:

• Vocabulary size. Larger vocabularies reduce sequence length (cheaper attention) but increase the embedding table. 32k suits English-dominant models; 100k+ handles multilingual corpora.
• Digit tokenisation. Splitting "1234" as ["1","2","3","4"] vs. ["12","34"] affects arithmetic reasoning. Recent pipelines force single-digit tokenisation.
• Byte fallback. Sentencepiece's byte fallback ensures no character is out-of-vocabulary, at the cost of longer sequences for rare scripts.

Decontamination: Keeping Benchmarks Blind

A pipeline that leaks benchmark test questions into pretraining data produces inflated evaluation numbers that do not generalise. The standard approach is n-gram overlap detection: for each evaluation example, compute 13-gram overlap with the training corpus and flag or remove documents above a threshold.

This is harder than it sounds. Benchmark text appears in forum posts ("what is the answer to this MMLU question?"), in model cards, and in synthetic-data repositories. A rigorous pipeline runs decontamination after deduplication, against every evaluation suite the team plans to report.

When It Falls Down

Aggressive perplexity filtering removes minority languages. A KenLM trained on English Wikipedia assigns high perplexity to text in low-resource languages, silently gutting multilingual representation. RefinedWeb and similar pipelines address this by running separate per-language filters, but that requires per-language KenLMs and complicates the pipeline significantly.

MinHash misses semantic duplication. Two articles covering the same news event in different words pass MinHash easily. The model still memorises the event; the duplication is conceptual rather than lexical. There is no cheap fix; the closest solution is embedding-based deduplication, which is expensive at trillion-token scale.

Decontamination is never complete. Benchmark contamination detection relies on string overlap; paraphrased or translated benchmarks slip through. Models that perform surprisingly well on a specific benchmark should be scrutinised for contamination even after a decontamination pass.

Tokeniser training on a biased sample propagates bias. If the 100 GB tokeniser-training sample over-represents English news prose, the resulting vocabulary will be inefficient for code, maths notation, and non-Latin scripts, even if these domains appear in the final training mix.

Mixture weights interact non-linearly with downstream tasks. A 1% increase in code data can improve maths performance by 5-10% through indirect reasoning transfer. These interactions are not captured by perplexity-based ablations alone, requiring expensive downstream evaluations to detect.

Further Reading

• Penedo et al. (2023), "The RefinedWeb Dataset for Falcon LLM": https://arxiv.org/abs/2306.01116 - the most thorough published treatment of a production web pipeline with quantitative ablations.
• Lee et al. (2022), "Deduplicating Training Data Makes Language Models Better": https://arxiv.org/abs/2107.06499 - canonical reference for MinHash and suffix-array deduplication at scale.
• Hoffmann et al. (2022), "Training Compute-Optimal Large Language Models" (Chinchilla): https://arxiv.org/abs/2203.15556 - establishes the token-budget framing that motivates corpus scale decisions.
• Gao et al. (2021), "The Pile: An 800GB Dataset of Diverse Text for Language Modeling": https://arxiv.org/abs/2101.00027 - the canonical multi-domain mixture reference with documented composition rationale.