Pretraining Data Pipelines
Web-scale corpus construction, filtering, deduplication, decontamination and data mixtures.
20concepts
140flashcards
152minutes of reading
- 01 Code Data Curation A systematic account of how raw source code from the internet is transformed into a deduplicated, filtered, mixed, and decontaminated pretraining corpus for code-focused language models.
- 02 Data Mixtures and Domain Weighting Domain weighting determines how much of each data source a model sees during pretraining, and getting this wrong can cost tens of thousands of GPU-hours or silently cripple downstream task performance.
- 03 Data Pipelines at Scale Building a pretraining corpus requires extracting, filtering, deduplicating, and mixing hundreds of billions of tokens from heterogeneous sources while keeping benchmark contamination out.
- 04 Web-Scale Corpus Construction Building a pretraining corpus at web scale requires five tightly coupled stages - extraction, quality filtering, deduplication, source mixing, and decontamination - each of which silently determines what a model can and cannot know.