The Mid-Training Stage and Data Annealing

Annealing the last 40 million tokens of Llama 3 8B pretraining, with the learning rate going linearly to zero and the data mixture upsampled toward very high quality sources, improved GSM8K validation performance by 24.0% and MATH by 6.4%. On the 405B model the same intervention was negligible (Grattafiori et al., 2024, The Llama 3 Herd of Models, arXiv:2407.21783). Two facts worth sitting with: a rounding error's worth of tokens moved a benchmark by a quarter, and the effect disappeared at scale.

That is the mid-training stage. It is not fine-tuning, because the objective is still next-token prediction over a broad corpus. It is not pretraining, because the mixture is curated rather than web-scale and the learning rate is on its way down. It is a distinct phase with its own rules, and as of 2026 essentially every published frontier recipe has one.

What the stage actually does

Three jobs, usually at once.

Capability patching. Pretraining corpora are dominated by whatever the web contains, which is not mathematics or code in the proportions you want. Mid-training upsamples the thin domains while the model still has enough learning rate to move. OLMo 2 formalised this: after stage-one pretraining, a stage-two mid-training phase consumes the remaining 5% to 10% of the FLOP budget on Dolmino Mix 1124, a curated blend of high-quality web, math and domain sources, built at 50B, 100B and 300B token budgets (OLMo Team, 2025, 2 OLMo 2 Furious, arXiv:2501.00656).

Long-context extension. Raising the context window is a mid-training operation because it needs a real token budget on genuinely long documents and a positional-encoding change, not a few thousand supervised examples. Code Llama's long-context fine-tuning stage used 20B tokens for exactly this (Rozière et al., 2023, arXiv:2308.12950).

Format priming. Exposing the model to the shapes it will be asked to produce later (chain-of-thought traces, tool-call syntax, structured outputs) while the learning rate is still non-trivial makes the subsequent post-training stage cheaper and more stable.

Annealing as a measurement instrument

The more interesting use of annealing is diagnostic. Because the effect of a small dataset is amplified during the decay phase, you can use a short anneal to score a data source before committing it to a full run. Llama 3 did this by taking a 50%-trained 8B checkpoint, annealing the learning rate to zero over 40B tokens with 30% weight on the candidate dataset and 70% on the default mix, and reading off the benchmark delta (Grattafiori et al., 2024, arXiv:2407.21783).

OLMo 2 runs the same idea under the name microannealing, and reports something that makes the technique far more useful than it first appears: sources assessed independently this way retain their effectiveness when mixed with the rest of the corpus (OLMo Team, 2025, arXiv:2501.00656). Additivity is not guaranteed a priori, and without it every candidate mixture would need its own full run.

The quantity on the left is cheap, comparable across candidates, and correlated with what the source contributes in a real run. That is the whole argument for the technique.

Why the decay phase is special

The mechanism is not fully settled, but the shape of it is consistent across reports. During the high, stable portion of the schedule the model is exploring; individual batches contribute noise that later batches overwrite. As the learning rate decays, updates stop being overwritten and the model commits. Data seen during commitment is therefore weighted far more heavily per token than data seen during exploration.

This is also why WSD schedules and annealing are the same conversation. The stable phase of a WSD run is the exploration regime, the short decay is the commitment regime, and mid-training is the decision about what the model commits to (Hu et al., 2024, MiniCPM, arXiv:2404.06395).

When it breaks

The Llama 3 result contains its own warning: the 405B model gained nothing from annealing on math data, because it already had the capability. Mid-training patches deficits. If there is no deficit, the tokens are wasted, and if the deficit is caused by something other than data scarcity (a tokeniser that shreds numerals, a context window too short for the task) then adding data does not address it.

Over-curating is the other failure. A mid-training mixture heavily weighted toward benchmark-adjacent data produces a model that looks better than it is, and the decay-phase amplification that makes annealing powerful also makes contamination through this channel unusually damaging. Decontaminate the mid-training mix harder than the pretraining mix, not less.