Filter Strength Versus Token Yield

A quality filter has one knob and two effects. Turning it up raises the average quality of what remains and lowers how much remains. The published evidence points in both directions, and the apparent contradiction dissolves once you notice that the two camps are operating under different constraints.

Two regimes, two answers

When unique data is plentiful relative to compute, filtering harder wins, and by large margins. DataComp-LM's central result is that model-based filtering is the dominant lever in dataset design: their DCLM-Baseline, about 2.6 trillion tokens selected from a 240 trillion token pool, trained a 7 billion parameter model to roughly 63 to 64 percent 5-shot MMLU, a figure that differs slightly between the arXiv and proceedings versions (Li et al., 2024, arXiv:2406.11794). FineWeb-Edu makes the same point at a different threshold: 1.3 trillion tokens kept out of 15 trillion, with sharp gains on MMLU and ARC (Penedo et al., 2024, arXiv:2406.17557).

When unique data is scarce relative to compute, the sign flips. The data-constrained scaling study tested exactly this and found that relaxing commonly used filters, including perplexity filtering and some deduplication, can be the better move, because the tokens a strict filter discards are tokens you will otherwise have to manufacture by repetition (Muennighoff et al., 2023, arXiv:2305.16264).

The reconciliation is that filter strength is not a property of the data. It is a joint decision with the compute budget and the epoch count, and the quantity being optimised is the quality of the effective corpus the model actually sees, repetitions included.

How to set the knob

The decision procedure that follows from this is mechanical.

Fix the compute budget and the model size, which fixes the number of training tokens \(D\). Measure the unique yield \(U\) at several threshold settings. For each setting, compute the implied epoch count \(E = D/U\) and the effective data under the geometric-decay model. Then run a short ablation at each setting, matched on training tokens rather than on dataset size, and pick the threshold with the best result.

Matching on training tokens is the part teams get wrong. Comparing a strict filter's corpus against a permissive one at equal dataset size compares two different compute budgets and tells you nothing about the choice you are actually making.

When it breaks

A classifier filter imports its reference corpus's biases. A model-based filter scores documents by resemblance to a curated reference set, so it inherits that set's language, register and subject distribution. The resulting corpus is high quality in the reference's terms, which may exclude dialects, technical registers or languages the final model needs.

Filters interact with the benchmarks used to tune them. A threshold chosen by MMLU gain selects for encyclopaedic, exam-shaped text. That is a legitimate target, and it is not the same as selecting for text that makes a good conversational or coding model, so the ablation suite is part of the filter design.

Short ablations mislead about long runs. Ablations at a few hundred billion tokens can favour a corpus that saturates later. The data-constrained regime is precisely where long-horizon behaviour diverges from short-horizon behaviour, because repetition only begins to bite after the first pass.

The discarded pile is not waste. Low-scoring documents can be useful as the base for synthetic rewriting, as curriculum material early in training, or as coverage for rare topics. Deleting rather than tiering them throws away optionality that costs only storage to keep.