Code Data Curation

Roughly 86 % of GitHub repositories are forks, and a substantial fraction of the remainder contain auto-generated boilerplate, minified JavaScript, and vendored dependencies. Feed that raw crawl into a language model and you train primarily on noise. The Stack v2 (used for StarCoder2) grew to 4× the size of its predecessor not by relaxing quality standards but by finding better signals for what "quality" means in code. The discipline that bridges raw crawl and a useful pretraining corpus is code data curation, and its decisions propagate directly into downstream coding ability.

Extraction and Language Detection

The first bottleneck is getting source code out of large web crawls or version-control archives in a reproducible way. Software Heritage provides persistent, content-addressed snapshots of public repositories under stable identifiers (SWHIDs), which StarCoder2 adopted to make its corpus fully auditable. Common alternatives start from GitHub archive dumps, BigQuery public datasets, or Google's Common Crawl filtered for code-like MIME types.

Once files are obtained, language identification is non-trivial. File extensions cover the common cases (.py, .rs, .java) but miss polyglot files, template languages, and configuration DSLs. Downstream tools typically combine extension heuristics with libraries like linguist (GitHub's language detector) and a byte-level trigram classifier as a fallback.

A useful early filter is file size. Files below roughly 100 bytes are usually empty or stubs; files above a few megabytes are usually minified assets or auto-generated serialised data. Both categories have near-zero signal-to-noise ratio and are cheap to remove:

MIN_BYTES = 100
MAX_BYTES = 1_048_576  # 1 MiB

def passes_size_filter(content: bytes) -> bool:
    return MIN_BYTES <= len(content) <= MAX_BYTES

Quality Filtering

Quality filtering for code differs structurally from natural-language filtering because "quality" is partly objective: code either compiles or it does not, has well-formed syntax or it does not, follows a consistent style or it does not.

Heuristic filters that the BigCode and DeepSeek-Coder teams converged on include:

Star-count filtering is an attractive proxy for quality (popular repositories presumably contain better code) but is empirically unreliable. SantaCoder found that restricting to repositories with 5+ GitHub stars degraded performance, likely because star counts correlate with project age and novelty rather than code quality, and because obscure but well-written codebases are eliminated. This is a consistent lesson: social signals are weak quality proxies in code.

A stronger signal is compiler or interpreter feedback. DeepSeek-Coder used compiler-based filtering for statically typed languages (Java, C, C++) to reject files that could not be parsed. For Python, ast.parse provides a syntax check at low cost.

Repository-level context also matters. Keeping files alongside their README, tests/, and requirements.txt gives the model richer context than individual-file extraction. StarCoder2 explicitly incorporated pull-request bodies and Kaggle notebooks to add natural-language-to-code linkage.

Near-Duplicate Detection and Deduplication

Deduplication is perhaps the highest-leverage single operation in corpus construction. Lee et al. (2021) showed that deduplicated training data reduces verbatim memorisation by roughly 10× and that models converge faster because they are not repeatedly exposed to the same examples. On the C4 dataset, one 61-word sentence appeared more than 60,000 times; equivalent redundancy exists at scale in code (think auto-generated files, copy-pasted boilerplate, and widely-forked repositories).

Two approaches dominate:

MinHash LSH operates at the document level. Each file is shingled into overlapping n-grams (typically 5-grams of tokens), a MinHash signature of fixed width (e.g., 128 or 256 hash functions) is computed, and Locality Sensitive Hashing groups candidate near-duplicates. Two files whose Jaccard similarity exceeds a threshold (commonly 0.7-0.8) are considered near-duplicates; all but one are discarded.

Suffix array exact substring deduplication finds shared substrings of length >= L across all documents. It detects copy-paste at sub-file granularity: the same 50-line function vendored into 400 repositories can be identified without hashing the whole file. This is computationally heavier but catches partial duplicates that MinHash misses.

In practice, code corpora run MinHash first (cheap, scales to trillions of tokens) and apply suffix arrays to the residual for high-value languages where boilerplate is a known problem.

An important subtlety for code: deduplication should operate per-language, not across languages. A Python snippet that is syntactically identical to a Ruby snippet carries distinct semantics and should not be collapsed.

Corpus Mixing and Data Schedules

Pretraining on only code produces models that struggle with natural-language docstrings, comments, and commit messages. The reverse (only natural language) misses syntax. The optimal mix is neither extreme, and it changes across training.

Llama 3 and Code Llama both used a two-phase approach: a general pretraining phase on a mixed web + code corpus, followed by a code-heavy "annealing" phase (Code Llama's long-context warmup used 500B additional code tokens). This phased strategy lets the model absorb world knowledge first, then specialise.

Within the code slice itself, language weighting requires care. Raw GitHub statistics are dominated by JavaScript and Python; smaller languages (Rust, Haskell, OCaml) are underrepresented by orders of magnitude. Oversampling rare languages above their natural frequency is standard practice, though the optimal multipliers are derived empirically or via small ablation runs rather than from theory.

A representative mixing pseudocode:

LANGUAGE_WEIGHTS = {
    "python": 1.0,
    "javascript": 0.8,
    "java": 0.6,
    "rust": 3.0,      # oversampled
    "haskell": 5.0,   # severely oversampled
    ...
}

def sample_batch(pool, weights, batch_size):
    return weighted_sample(pool, weights, n=batch_size)

Natural language is typically held at 10-30 % of the code-phase corpus to preserve instruction-following and comment comprehension.

Tokeniser Training and Code Alignment

The tokeniser is trained on the corpus, not before it, and its vocabulary directly affects compute efficiency. A tokeniser trained on general web text tends to split identifiers like camelCaseFunction into many fragments, increasing sequence length and wasting context window capacity.

Code-specialised tokenisers address this by:

• Training byte-pair encoding (BPE) on a representative sample of the actual code corpus (not Wikipedia + books).
• Setting vocabulary size larger than typical NL tokenisers (32k-100k tokens) to improve coverage of identifier fragments, operators, and indentation patterns.
• Preserving whitespace explicitly. Python indentation is load-bearing; collapsing whitespace during tokenisation destroys structural information.
• Reserving special tokens for fill-in-the-middle (FIM) tasks: <|fim_prefix|>, <|fim_middle|>, <|fim_suffix|>.

A tokeniser trained on code achieves materially shorter sequences for the same file, meaning the same training budget covers more semantic content.

Decontamination

A training corpus that contains solutions to HumanEval, MBPP, or LeetCode problems will produce inflated benchmark scores. The problem is subtle: because code corpora are drawn from GitHub, and GitHub contains users' solutions to these well-known benchmarks, overlap is essentially guaranteed unless actively removed.

Decontamination involves:

1. Canonicalising both the benchmark test cases and the corpus documents (strip comments, normalise whitespace, lowercase).
2. Computing n-gram overlap (typically 10-gram or longer substring matches) between each benchmark problem and each training document.
3. Removing any training document exceeding an overlap threshold.

The BigCode team published their decontamination tooling alongside The Stack to make this reproducible. This is not optional hygiene: without it, benchmark comparisons across models trained on different corpora are not meaningful.

When it Falls Down

Licence laundering. Filters that accept only "permissive" licences (MIT, Apache 2.0) are easier to declare than to enforce. GPL code hosted in repositories that omit or mis-tag the licence slips through. Several models have faced legal challenges on these grounds.

Language creep. A file extension filter trained on GitHub statistics will badly underestimate DSLs, templating languages (Jinja2, ERB), and configuration languages (Nix, HCL) that are increasingly present in modern repositories. These files often pass quality filters and inject noise.

Deduplication brittleness. MinHash is sensitive to tokenisation. A file with CRLF line endings deduplicated against the same file with LF endings may not be identified as a duplicate if shingling operates on raw bytes. Normalise line endings before hashing.

Benchmark contamination drift. New benchmarks (HumanEval+, LiveCodeBench) are added after training. A corpus decontaminated against HumanEval may still contain solutions to its successors. Decontamination is necessarily retrospective.

Oversampling instability. Aggressively oversampling rare languages can cause training instability if the language's loss landscape is significantly different. Monitoring per-language perplexity during training catches this early; ignoring it produces unexpectedly poor performance on majority languages.

Compiler-filter brittleness at scale. Running javac or gcc on every file in a multi-trillion-token corpus is computationally prohibitive. Approximations (AST parsing, regex-based syntax checks) catch only a fraction of malformed files.

Further Reading

• Kocetkov et al., "The Stack: 3 TB of permissively licensed source code," BigCode, 2022. https://arxiv.org/abs/2211.15533
• Lee et al., "Deduplicating Training Data Makes Language Models Better," ACL 2022. https://arxiv.org/abs/2107.06499
• Lozhkov et al., "StarCoder2 and The Stack v2: The Next Generation," BigCode, 2024. https://arxiv.org/abs/2402.19173
• Guo et al., "DeepSeek-Coder: When the Large Language Model Meets Programming," 2024. https://arxiv.org/abs/2401.14196