Pretraining Data Pipelines
Web-scale corpus construction, filtering, deduplication, decontamination and data mixtures.
20concepts
140flashcards
152minutes of reading
- 01 Benchmark Decontamination Benchmark decontamination is the process of identifying and removing evaluation set examples from a model's pretraining corpus so that reported benchmark scores reflect genuine generalisation rather than memorised answers.
- 02 Curriculum and Data Ordering How the sequence and mixture proportions of training batches affect what an LLM learns and when, and why naive i.i.d. sampling often leaves capability on the table.
- 03 Data Provenance and Licensing Data provenance tracks the origin and transformation history of every byte in a training corpus, while licensing determines whether you are legally permitted to use it at all.
- 04 Deduplication and Memorisation Duplicate training examples cause language models to memorise and verbatim-reproduce training text, so removing duplicates before training reduces regurgitation, improves generalisation, and lowers compute waste.
- 05 Document-Level vs Token-Level Deduplication Document-level deduplication removes whole near-duplicate pages using hashing, while token-level deduplication removes repeated spans within and across documents using suffix arrays, each with distinct cost-quality trade-offs in web-scale corpus construction.
- 06 Exact and Fuzzy Deduplication How hash-based exact matching and MinHash locality-sensitive hashing together remove the duplicate content that otherwise inflates memorisation and wastes training compute.
- 07 Heuristic Quality Filters Heuristic quality filters are rule-based passes over raw web text that remove boilerplate, malformed documents, and low-information content before any classifier is trained or applied.
- 08 Language Identification and Filtering Language identification assigns a language label to each document in a web crawl so pipelines can retain target-language text and discard everything else before quality filtering begins.
- 09 MinHash and LSH for Deduplication MinHash estimates Jaccard similarity between documents in constant memory, and LSH bucketing turns that estimate into a sub-linear nearest-neighbour search, making corpus-scale near-deduplication tractable.
- 10 Multilingual Data Balancing Multilingual data balancing decides how many tokens from each language land in a pretraining corpus, trading raw web proportions against deliberate upsampling to serve low-resource languages without degrading high-resource ones.
- 11 PII Detection and Removal Scrubbing personally identifiable information from web-scale corpora before LLM pretraining reduces memorisation risk and legal exposure, but every detection method trades recall against corpus damage.
- 12 Quality Filtering with Classifiers Classifier-based quality filtering uses lightweight models trained on curated reference corpora to score and discard low-quality web documents before LLM pretraining.
- 13 Text Extraction and Boilerplate Removal Converting raw HTML from web crawls into clean, main-content text is a lossy signal-recovery problem, and the choices made here propagate irreversibly through every downstream filtering and training stage.
- 14 Toxicity Filtering of Pretraining Data Toxicity filtering removes hate speech, obscene language, and harmful content from web-crawled corpora before LLM pretraining, using blocklists, classifiers, and API-based scoring, each with distinct recall/precision trade-offs and equity side effects.
- 15 Training a Tokeniser on a Corpus Training a tokeniser on your pretraining corpus, rather than borrowing one designed for another model, directly controls sequence length, vocabulary coverage, and how the model sees numbers, code, and non-English text.
- 16 Vocabulary Size Trade-offs Choosing a tokeniser vocabulary size forces a three-way tension between sequence length, embedding table memory, and coverage of rare or multilingual text.