The Training Run That Never Ends: Continued Pretraining, Mid-Training and Model Growth

During the final 40 million tokens of Llama 3 8B's pretraining, Meta annealed the learning rate linearly to zero and upsampled the data mixture toward very high quality sources. GSM8K validation performance rose 24.0%; MATH rose 6.4%. On the 405B model, the identical intervention did essentially nothing (Grattafiori et al., 2024, The Llama 3 Herd of Models, arXiv:2407.21783).

Forty million tokens is 0.0003% of the 15 trillion that model saw. Something about the end of a training run makes tokens worth orders of magnitude more than they are in the middle, and that observation has quietly restructured how frontier models get built. The two-stage picture everyone learned, pretrain then post-train, now has a third stage wedged between them, and a fourth operation that does not fit the sequence at all: reaching into a finished checkpoint and changing its shape.

Why this matters: A pretrained checkpoint is no longer an artefact you finish and ship. It is a starting condition you keep editing: continuing its pretraining on new data, annealing it toward capabilities it lacks, stacking its layers to make it deeper, or cloning its feed-forward blocks into experts. Each of those operations obeys rules that are different from, and sometimes opposite to, the rules of pretraining. Teams that apply pretraining instincts to this stage burn budget and kill runs that were working.

TL;DR

• Resuming a finished pretraining run fails for a boring mechanical reason: the checkpoint sits at a decayed learning rate near zero, so new gradients are multiplied by almost nothing. You have to deliberately re-warm, and re-warming makes loss worse before it makes it better.
• The complete recipe is three knobs. Re-warm the learning rate, re-decay it over the continued run, replay a small fraction of the original corpus. Ibrahim et al. showed this matches full retraining from scratch on both old and new distributions, at 405M and again at 10B parameters (arXiv:2403.08763).
• Replay's famous 5% is a measurement, not a law. At 100B tokens per language, a 25% replay rate turned out to be a better use of compute than spending the same compute on a bigger model, while 50% was not (Abbes et al., 2025, arXiv:2508.01908).
• The performance dip at the start of continued pretraining is a named phenomenon with a predictable shape, not a misconfiguration. Runs killed at 2,000 steps are usually killed inside the stability gap (Guo et al., 2024, arXiv:2406.14833).
• Annealing doubles as a measurement instrument. A short decay over a candidate dataset scores that dataset cheaply, and OLMo 2 reports the scores stay valid when sources are mixed, which is what makes the technique usable at all (OLMo Team, 2025, arXiv:2501.00656).
• Of four atomic model-growth operators evaluated head to head at LLM scale, the simplest won: depthwise stacking (Du et al., 2024, NeurIPS, arXiv:2405.15319). Growth buys a better initialisation, not a better ceiling.
• Upcycling a dense checkpoint into a mixture of experts is a trade of memory for accuracy at fixed training compute. Nemotron-4 15B upcycled over 1T tokens reached 67.6% MMLU against 65.3% for continuing the dense model on the same tokens (He et al., 2024, arXiv:2410.07524).

At a Glance

flowchart LR
    A[Web-scale corpus] --> B["Stage 1 pretraining<br/>stable high LR"]
    B --> C["Stage 2 mid-training<br/>curated mix and decaying LR"]
    C --> D["Post-training<br/>SFT and RL"]
    B -.checkpoint reused.-> E["Continued pretraining<br/>new domain and re-warmed LR"]
    B -.checkpoint reshaped.-> F["Model growth<br/>or MoE upcycling"]
    E --> C
    F --> C
    D --> G[Deployed model]
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    class A,B slate
    class C purple
    class E teal
    class F amber
    class D,G emerald

The dotted edges are the point. A stage-one checkpoint is not consumed by the pipeline; it is a branch point that several different operations can start from, each producing something that rejoins the same mid-training and post-training tail.

Before the Checkpoint Was Reusable

For most of the deep learning era, "continue training this model on new data" meant fine-tuning, and fine-tuning meant a small supervised dataset, a low learning rate and an acceptance that you would damage general capability in exchange for task performance. The continual learning literature spent a decade building machinery to avoid that damage. Elastic weight consolidation, the best-known result, penalises movement in parameters that mattered to earlier tasks, weighted by the diagonal of the Fisher information matrix, which falls out of a Laplace approximation to the previous-task posterior (Kirkpatrick et al., 2017, PNAS 114:3521-3526).

That machinery largely did not survive contact with language models at scale. Not because it was wrong, but because the simplest alternative, mixing old data back in, turned out to work better and cost less. What changed was the economics: once a pretraining run costs eight figures, the question stops being "how do we avoid forgetting" and becomes "how do we avoid ever paying that again".

timeline
    title From fine-tuning to a reusable checkpoint
    2016 : Net2Net establishes function-preserving width and depth growth
    2017 : Elastic weight consolidation frames forgetting as Bayesian updating
    2022 : Sparse Upcycling turns dense checkpoints into mixtures of experts
    2023 : Code Llama continues Llama 2 for 500B code-heavy tokens
         : Gupta et al. measure what learning-rate re-warming actually does
         : SOLAR 10.7B ships depth up-scaling as a production recipe
    2024 : Ibrahim et al. show re-warm plus re-decay plus replay matches retraining
         : MiniCPM introduces warmup-stable-decay for open-ended runs
         : Llama 3 uses a 40M-token anneal to both patch and measure
         : G_stack wins the head-to-head model-growth comparison
    2025 : OLMo 2 names mid-training as a stage and publishes its mixture
         : Replay scaling gets re-measured at 100B tokens per language
    2026 : Optimal continued-pretraining hyperparameters get their own scaling laws

The 2023 to 2024 cluster is where the stage became legible. Code Llama demonstrated that 500B additional tokens on top of a 2T-token base, 85% of them code, produced large gains on HumanEval and MBPP, and that a further 20B-token long-context stage could move the usable window to 16k with improvements out to 100k (Rozière et al., 2023, Code Llama, arXiv:2308.12950). Llemma repeated the pattern one level up, continuing Code Llama on Proof-Pile-2 to beat every open base model on MATH at equal parameter count (Azerbayev et al., 2024, ICLR, arXiv:2310.10631). The checkpoint had become a substrate.

[IMAGE: Two stacked horizontal bars showing token budgets to scale. Top bar: Llama 2 pretraining, 2T tokens, in slate. Bottom bar: the same 2T in slate plus a 500B code segment in teal and a 20B long-context segment in amber, labelled Code Llama. Caption: "Code Llama added 26% more tokens to a finished model and changed what it was for."]

What Actually Happens When You Resume a Run

The terminal learning rate problem

Almost every pretraining recipe ends by annealing the learning rate toward zero. That final decay is not a formality; it is where a large share of the total loss reduction lands, and it leaves the weights in a sharp, well-fitted basin. Resume from there and the update rule gives you

so the gradient from the new distribution, however informative, is multiplied by nothing. The model sees the new corpus and does not move. This is the first thing that goes wrong, and it is completely mechanical.

The fix is to raise the learning rate again, and the consequence of the fix is the second thing that goes wrong. Gupta et al. continued models pretrained on the Pile (300B tokens) onto SlimPajama (297B tokens) under a range of warm-up strategies and found that re-warming increases loss on both the upstream and the downstream distribution at first. Over a long enough run, the re-warmed models won anyway, outperforming models trained from scratch even for a large downstream dataset (Gupta et al., 2023, arXiv:2308.04014).

Three knobs, and why the third one is a purchase

[IMAGE: Three loss curves on shared axes over continued-pretraining steps, all starting from the same decayed checkpoint. Curve 1 (no re-warm) stays flat. Curve 2 (re-warm, no replay) dips then falls below baseline on the new data while upstream loss climbs off the top of the frame. Curve 3 (re-warm plus re-decay plus replay) dips deepest early and ends lowest on both. Caption: "The recipe that wins is the one that looks worst at step 2,000."]

Ibrahim et al. reduced the practical question to a recipe. Re-warm the learning rate. Re-decay it over the course of the continued run. Replay a small fraction of the original corpus. With all three in place, continued pretraining matched a full retraining baseline on final loss and on language model evaluations, under a weak distribution shift (English to English) and a strong one (English to German), demonstrated at 405M parameters and confirmed at 10B (Ibrahim et al., 2024, TMLR, arXiv:2403.08763).

Re-decaying is the knob teams skip, and skipping it is expensive. A continued run held at a constant elevated learning rate never gets the commitment phase; its training loss looks fine while its evaluation numbers lag. Treat the continued run as a complete schedule in miniature.

Replay is the knob teams misunderstand. It is not a free defence against forgetting; it is a purchase. At replay fraction \(r\), a fraction \(r\) of your continued-pretraining compute is spent on data the model has already seen. The 5% figure that circulates as a default came from a specific scale and a specific shift strength. The more recent measurement, across languages at 100B tokens per language on Llama-family architectures, found that a 25% replay rate is a more compute-efficient use of budget than putting the same compute into a larger model, while 50% is not (Abbes et al., 2025, arXiv:2508.01908). Read that as a broad, forgiving optimum between roughly 5% and 25%, with the right point determined by how far the new distribution sits from the old.

That same work supplies the one refinement of rehearsal with a real case at this scale: gradient alignment, which asks whether the replay gradient and the new-data gradient point in compatible directions and corrects when they conflict, implemented with negligible compute and memory overhead over plain replay.

How high to re-warm

This is where practice has been thinnest. Common guidance puts the re-warm target somewhere between 10% and 50% of the original peak learning rate, scaled down as the shift weakens and as the continued budget shrinks. Recent work argues the guess is unnecessary: optimal continued-pretraining hyperparameters follow stable, predictable scaling laws. The proposed procedure fits functions from compute budget to optimal hyperparameters on small proxy models, then evaluates a checkpoint's validation loss to estimate its equivalent pretraining compute, and reads the hyperparameters off the inverse law, reportedly cutting search overhead by up to 90% (Zhou et al., 2026, arXiv:2606.05610).

The idea that a checkpoint has an "equivalent pretraining compute" readable from its loss is the conceptually interesting part. It says a checkpoint's position in training is a scalar you can recover, regardless of how it got there, which is exactly what you need if checkpoints are going to be traded, grown and resumed by people who did not train them.

Schedules that expect to be resumed

The deeper fix is to stop decaying to zero. Warmup-stable-decay warms up, holds a constant peak learning rate for the bulk of training, then decays sharply over a short final window. Because the stable phase encodes no total step count, every checkpoint taken during it is a legitimate resumption point, and you can branch a short decay off any of them to get a deployable model (Hu et al., 2024, MiniCPM, COLM, arXiv:2404.06395).

A cosine schedule commits you to a horizon on step one. WSD leaves the run open. That is a small change with a large downstream consequence, because it makes "the training run never ends" an architectural property of the schedule rather than a thing you improvise later.

[IMAGE: Line chart with two learning-rate curves over training steps. Curve A, cosine, rises and falls smoothly to zero at step N. Curve B, WSD, rises, holds flat, then drops steeply in the final 10%, with three dashed branch-off decays drawn from different points on the flat section. Caption: "Cosine commits to a horizon; WSD makes every stable-phase checkpoint a branch point."]

Seeing It in Motion

The lifecycle of a continued-pretraining run has a shape that is worth internalising, because the middle of it looks like failure.

stateDiagram-v2
    [*] --> Frozen: load decayed checkpoint
    Frozen --> Rewarming: raise LR over warmup steps
    Rewarming --> StabilityGap: disruption exceeds accumulation
    StabilityGap --> Recovery: new-data gradient accumulates
    StabilityGap --> Destroyed: LR pushed model out of basin
    Recovery --> Annealing: begin re-decay on curated mix
    Annealing --> [*]: deployable checkpoint
    Destroyed --> [*]: restart with gentler schedule

The branch out of StabilityGap is the decision nobody can make confidently in-flight, and it is discussed at length below. The other thing this diagram encodes is that Annealing is not a separate project from continued pretraining; it is the tail of the same schedule, which is why mid-training and CPT keep collapsing into one conversation.

The second flow worth drawing is the source-selection loop, because it is the operation that turns annealing from a technique into an instrument.

sequenceDiagram
    participant Ckpt as Half-trained checkpoint
    participant Mix as Default mixture
    participant Cand as Candidate source
    participant Anneal as Short anneal run
    participant Eval as Benchmark suite
    Ckpt->>Anneal: load frozen copy
    Mix->>Anneal: 70% of tokens
    Cand->>Anneal: 30% of tokens
    Anneal->>Eval: annealed model
    Eval-->>Cand: score delta vs mix-only baseline
    Note over Ckpt,Eval: Repeat per candidate from the same checkpoint;<br/>OLMo 2 reports the deltas stay valid when sources are combined

Running each candidate from the same frozen checkpoint is what makes the deltas comparable. Llama 3's version used a 50%-trained 8B checkpoint annealed to zero over 40B tokens with a 30/70 split between candidate and default mix (Grattafiori et al., 2024, arXiv:2407.21783). OLMo 2 runs the same idea as microannealing and reports the result that makes it practical: sources assessed independently retain their effectiveness when mixed with the rest of the corpus (OLMo Team, 2025, arXiv:2501.00656). Additivity is not free, and without it every candidate mixture would need its own full run.

Finally, the decision itself. Four operations start from a finished checkpoint, and choosing between them is mostly a question of what you are short of.

flowchart TB
    Q1{"What is missing?"}
    Q1 -->|Knowledge in a domain| CPT["Continued pretraining<br/>billions of tokens"]
    Q1 -->|A thin capability| MID["Mid-training anneal<br/>curated mixture"]
    Q1 -->|Raw capacity and compute bound| GROW[Depth or width growth]
    Q1 -->|Raw capacity and memory rich| UPC[MoE upcycling]
    Q1 -->|Behaviour or format| PT["Post-training<br/>SFT and RL"]
    CPT --> R{"Have the original corpus?"}
    R -->|Yes| OK[Replay 5 to 25 percent]
    R -->|No| PROXY["Proxy corpus with weaker guarantee"]
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    class CPT,MID teal
    class GROW,UPC amber
    class PT emerald
    class R,Q1 slate
    class PROXY rose

Watch It Run

By the Numbers

Every row below is an extra-training intervention applied to an already-trained checkpoint, sized against what the checkpoint cost in the first place.

Sources: Llama 3 (Grattafiori et al., 2024); OLMo 2 (OLMo Team, 2025); Code Llama (Rozière et al., 2023); stability gap (Guo et al., 2024); replay recipe (Ibrahim et al., 2024) and replay scaling (Abbes et al., 2025); upcycling (He et al., 2024); SOLAR (Kim et al., 2024); LiGO (Wang et al., 2023). The SOLAR row reports an architectural fact, not a controlled comparison; the paper's broader performance claims are the authors' own. Percentages relative to original pretraining are computed from published token counts and are approximate.

[IMAGE: Log-scale horizontal bar chart of "extra tokens as a fraction of original pretraining" for each row in the table above, spanning six orders of magnitude from Llama 3's anneal at 3e-6 to Code Llama's 0.25. Caption: "The interventions that moved benchmarks most are not the ones that spent the most tokens."]

A Concrete Example

[IMAGE: Sankey-style flow of a 100B-token continued-pretraining budget splitting into 85B new legal tokens (teal) and 15B replay tokens (slate), with the 8B microanneal probe branching off mid-stream in amber and the final 5B anneal tail highlighted in purple. Caption: "Where a 100-billion-token budget actually goes, including the 8% spent on finding out what to train on."]

You hold an 8B open-weights model pretrained on 15T tokens. You want it strong on legal text. You have 64 H100s for a week and a 100B-token legal corpus. Here is the whole plan, arithmetic included.

Step 1 — Size the intervention. 100B on top of 15T is 0.67% of the original token budget. That is well into continued-pretraining territory rather than fine-tuning, and large enough that a real re-warm is justified.

Step 2 — Price it. Training FLOPs are about \(6ND\) for \(N\) parameters and \(D\) tokens:

An H100 SXM does roughly 495 TFLOP/s dense BF16 at peak. Assume 40% model FLOPs utilisation, which is a reasonable planning figure for a well-tuned 8B run and is an assumption, not a measurement: about 198 TFLOP/s per GPU, 12.7 PFLOP/s across 64. That gives

Retraining from scratch on 15T tokens would be \(6 \times 8\times10^9 \times 1.5\times10^{13} = 7.2 \times 10^{23}\) FLOPs, 150 times more, or about 22 months on the same cluster. That ratio is the entire economic argument for this stage.

Step 3 — Choose the replay fraction. Legal English against general English is a moderate shift, not English-to-German. Take \(r = 0.15\): 85B legal tokens, 15B replay. You do not have Meta's corpus, so replay comes from an open general-web proxy, which is weaker and worth flagging in the model card.

Step 4 — Lay out the schedule. Sequence length 8,192, global batch 4M tokens, so \(10^{11}/(4\times10^{6}) = 25{,}000\) optimiser steps. Original peak learning rate was \(3\times10^{-4}\), and the released checkpoint sits near \(3\times10^{-5}\). Re-warm to 30% of peak, \(9\times10^{-5}\), over 250 steps. Hold for 20,000. Decay to zero over the final 4,750.

Step 5 — Set the kill criterion before you start. Expect the stability gap. Legal benchmark accuracy should fall for the first few thousand steps. Recovery within the first 5-10% of the continued budget means steps 1,250 to 2,500, so the honest cut-off is around step 5,000, not step 2,000. Write that number down now, because you will not want to at step 3,000.

Step 6 — Spend 8% of the budget on measurement. Pause at step 12,500 and microanneal four candidate legal sub-corpora from that same frozen checkpoint, 2B tokens each at a 30/70 split. Suppose the deltas on your held-out legal QA set come back:

Eight billion tokens, 8% of the run, bought the information that the largest available source in the pile (marketing pages, 25B tokens) is actively harmful, and that a small one (contract templates, 8B) is worth upsampling. Without that measurement the obvious move is to weight by corpus size, which here would have been exactly wrong.

Step 7 — Build the anneal tail. Reserve the final 5B tokens, the decay phase, for the mixture the microanneals endorsed: court opinions and statutes upsampled, marketing pages dropped, contract templates weighted above their natural share, replay held at 15% throughout. Decontaminate this 5B harder than you decontaminated the other 95B, because decay-phase data is amplified and contamination entering here moves scores more than it should.

Step 8 — Report honestly. Two curves from step zero, legal benchmark and general benchmark, on the same cadence. The general curve will dip too. If it has not recovered to within a point or two of baseline by the end of the anneal, your replay fraction was too low and the next run uses 0.25.

Where It Breaks

The dip that never ends

[IMAGE: Two target-domain accuracy curves over the first 8,000 steps of continued pretraining, both dipping below the starting line. The recoverable run (emerald) bottoms out near step 2,000 and crosses back above baseline by step 4,500. The destroyed run (rose) keeps descending on a shallower slope with no inflection. A vertical dashed line at step 2,000 is labelled "where teams cancel". Caption: "Both curves look identical at the moment most people decide."]

The stability gap is a transient, except when it is not. If the re-warmed learning rate is high enough to push the model out of its basin, there is no recovery phase, only a slow re-descent from a much worse starting point, and the run is strictly worse than having started over with a gentler schedule. The genuinely hard part is that a deep-but-recoverable dip and a destroyed checkpoint look identical for thousands of steps. There is no in-flight diagnostic that reliably separates them, which is the strongest argument for conservative re-warming whenever the continued budget is small.

Guo et al.'s mitigations attack the duration rather than the existence of the gap, and one of them inverts standard pretraining practice: at fixed budget, multiple epochs over a well-chosen subset recovers performance faster than a single pass over the full corpus. Repetition is normally something pretraining avoids for memorisation reasons. At CPT scale with a modest budget, faster recovery is worth more than marginal diversity. Combined with filtering to a high-quality sub-corpus and keeping the mixture close to the pretraining distribution, this took OpenLlama-3B from 36.2% to 40.7% average medical performance on 40% of the original budget, with general performance improving rather than degrading (Guo et al., 2024, arXiv:2406.14833).

Watching the wrong curve

Upstream validation loss during a strong-shift continued run can rise for a long stretch while the target capability improves. Any early-stopping rule keyed to the old distribution terminates every successful run. The inverse failure is subtler and worse: with a small replay fraction and a shift too strong for the budget, surface fluency on the old distribution can hold steady while the capabilities it supported erode underneath. Perplexity does not move, the dashboard stays green, and the model quietly gets worse at things nobody is measuring.

Replay without a corpus

Every published replay result assumes access to the original pretraining data. For a checkpoint you acquired rather than trained, which is the common case, you do not have it, and licensing changes, deletion requests and infrastructure rot mean you may not have it even for models you did train. A proxy corpus is the standard substitute and it is genuinely weaker: you are replaying something adjacent to what the model learned rather than what it learned. Nobody has published a satisfying answer to this, and it is the largest gap between the literature and ordinary practice.

Contamination through the amplifier

The decay phase weights data far more heavily per token than the stable phase. That is why annealing works, and it is also why contamination entering through the mid-training mix does disproportionate damage. A benchmark-adjacent dataset that would be diluted to irrelevance in pretraining can move scores several points when introduced during commitment. The correct response is asymmetric: decontaminate the mid-training mixture harder than the pretraining mixture, not with the same rigour and certainly not with less.

There is a governance version of this too. Mid-training gives a lab a cheap, late-stage, benchmark-sensitive lever, applied after most external scrutiny of the data pipeline has happened. That is not an accusation, it is a structural observation about where the incentive sits.

Growth inherits everything

A grown model inherits its parent's tokeniser, positional-encoding scheme and base frequency, learned biases, corpus contamination, and failure modes. If the 7B parent was weak at multi-digit arithmetic because its tokeniser split numerals badly, the 11B child starts weak at multi-digit arithmetic for exactly the same reason. Depth stacking also hands you whatever aspect ratio the arithmetic produces rather than one anybody chose, and depth is serial at inference, so a stacked model can be meaningfully slower per token than a well-shaped model at the same parameter count.

The seam is the specific failure. SOLAR's depth up-scaling duplicates a 32-layer stack, then removes the last \(m = 8\) layers of the first copy and the first 8 of the second, concatenating \(2n - 2m = 48\) layers (Kim et al., 2024, arXiv:2312.15166). Cutting either side of the join is not arbitrary: a naive concatenation leaves the second copy's early layers, which expect embedding-like inputs, receiving deep representations instead. Where you cut determines whether the continued run recovers quickly or spends most of its budget repairing a discontinuity.

Upcycling's bill arrives at inference

At the instant of upcycling, every expert is identical, so the router's output carries no information and the gradient that would differentiate the clones is the same for all of them. Symmetry breaks only through random router initialisation and the load-balancing auxiliary loss, and the early steps are correspondingly delicate: too strong a balancing pressure and routing goes uniform, which reproduces the dense model at \(E\) times the parameter cost; too weak and the router collapses onto one copy. Skywork-MoE's use of per-layer adaptive auxiliary coefficients is a direct response, and implies the right pressure is not uniform across depth (Wei et al., 2024, arXiv:2406.06563).

The cost that surprises people is downstream. Skywork-13B upcycled into a 146B-parameter, 16-expert model: all \(E\) copies of every upcycled FFN must be resident even though a fraction is active per token. Reported slowdowns of up to roughly 40% for larger upcycled models come from routing overhead and increased active parameters. Sparsity reduces FLOPs; FLOPs are not the only thing that sets latency.

[IMAGE: Side-by-side memory footprint bars for a 13B dense model and its 16-expert upcycled descendant, with the active-parameters-per-token portion highlighted in teal and the resident-but-idle portion in amber. Caption: "Upcycling trades memory for accuracy. The idle bar still has to fit on the GPU."]

Alternative Designs

The honest summary of that table is that the rows are not competitors so much as answers to different questions. The one genuine rivalry is the last two against each other, and the deciding variable is whether you are short of compute or short of memory.

How It Is Used in Practice

The open record is unusually good here, because the labs that publish training details have converged on similar structures. OLMo 2 is the most explicit: stage one on a large web corpus, then a stage-two mid-training phase consuming the remaining 5-10% of the FLOP budget on Dolmino Mix 1124, a curated blend of high-quality web, math and domain sources, released at 50B, 100B and 300B token budgets so downstream users can pick (OLMo Team, 2025, arXiv:2501.00656). Publishing the mid-training mixture as a named artefact is itself a signal of how the stage is now regarded.

Meta's use of annealing as an evaluation instrument is the other pattern worth copying. The technique costs a fraction of a full run, produces comparable numbers across candidates, and, per OLMo 2's additivity finding, composes. For any team with more candidate data sources than budget, which is every team, this is the single most useful idea in this article.

On the growth side, the production examples are fewer and more cautious. SOLAR 10.7B is the clearest public depth up-scaling recipe; Skywork-MoE is the clearest public upcycling one. NVIDIA's upcycling work has been integrated into Megatron-Core, which matters more than the paper's benchmark numbers, because it moves the technique from "reproducible in principle" to "a flag in a training framework" (He et al., 2024, arXiv:2410.07524).

What none of this tells you is how the stage behaves after five or six successive rounds. Dohare et al.'s result on loss of plasticity, where standard deep learning degrades toward shallow-network performance over long task sequences (ImageNet binary classification falling from 89% on an early task to about 77% by the 2,000th), comes from a many-task continual setting rather than an LLM pipeline and should not be transferred uncritically (Dohare et al., 2024, Nature 632:768-774). It does, however, name a thing teams report anecdotally: diminishing gains per round that replay alone does not fix.

[IMAGE: Timeline strip showing one checkpoint lineage across four successive continued-pretraining rounds, with a small bar at each round showing the gain achieved, decreasing left to right, annotated "reported anecdotally; not yet measured at LLM scale". Caption: "Nobody has published what happens after the fifth round."]

Insights Worth Remembering

1. The last few percent of a training run is a different regime, not a smaller version of the same one. During the stable phase, individual batches contribute noise that later batches overwrite. During decay, updates stop being overwritten and the model commits. Data seen during commitment is weighted far more heavily per token, which is why 40M tokens moved GSM8K by 24%.

2. Re-warming is the price of adaptation, and the loss increase is the receipt. The same large steps that degrade a finely tuned solution are the only thing capable of reshaping it. A continued run whose target metric climbs monotonically from the first evaluation usually has a learning rate too low to be doing much.

3. Replay is a purchase, not a defence. At fraction \(r\), you are spending \(r\) of your budget on tokens the model has already seen, buying retention of the old distribution with tokens that would otherwise buy adaptation. Choose \(r\) by deciding how much upstream degradation you will accept, not by copying 5%.

4. The elegant continual-learning machinery lost to mixing old data back in. EWC constrains parameters through a Fisher-weighted penalty; replay supplies the true gradient of the old objective. At ten billion parameters the diagonal Fisher approximation is poor and expensive, and the old corpus is already on disk.

5. Annealing is worth more as a measurement instrument than as an intervention. A short decay over a candidate source, run from a frozen checkpoint, scores that source cheaply and comparably, and OLMo 2's additivity result means the scores survive composition. This converts data curation from guesswork into a small experiment.

6. Model growth buys a better initialisation, never a better ceiling. Every acceleration claim in this literature is properly phrased as "reaches loss \(L\) in fewer FLOPs". Many growth operators start below a from-scratch baseline and converge to a similar or worse final loss.

7. The simplest growth operator won. Four atomic operators evaluated head to head under one standardised setup, and depthwise stacking beat the learned and function-preserving alternatives (Du et al., 2024, arXiv:2405.15319). Sophistication has not paid off here yet.

8. Upcycling trades memory for accuracy at fixed training compute. A 2.3-point MMLU gain over continued dense training on the same tokens is real and modest, and it comes with \(E\) copies of every upcycled FFN resident in memory. Judge it on that trade, not the headline.

9. A checkpoint inherits its parent's mistakes at full strength. Tokeniser, positional encoding, contamination, and failure modes all copy forward, and upcycling copies them \(E\) times. Neither growth nor upcycling repairs a weak base; train it further first.

10. Schedule choice determines whether the run can be resumed at all. Cosine commits to a horizon on step one. WSD leaves the run open, and that single property is what makes "the training run never ends" a design decision rather than an improvisation.

Open Questions

How many rounds of continued pretraining can one lineage take? Every published result covers one or two rounds. The plasticity-loss literature measures degradation over thousands of tasks in a different setting, and the gap between those two regimes is unmapped. Whether a checkpoint continued five times behaves like a checkpoint continued once is, as of 2026, not something anyone has measured and published at LLM scale.

What is the right replay corpus when you do not have the original? Every result assumes access to the pretraining data. The proxy-corpus substitute is used constantly and evaluated almost never. An open question is whether proxy replay preserves the same properties as true replay, or only the ones we currently measure.

Can re-warm height be set from first principles? The scaling-law approach of estimating a checkpoint's equivalent pretraining compute from its validation loss and reading hyperparameters off a fitted curve is the most principled proposal so far (Zhou et al., 2026, arXiv:2606.05610), and it reports up to 90% less search overhead. Whether the laws hold across architectures and shift strengths that differ substantially from those fitted is not yet established.

Is there a diagnostic that separates a recoverable dip from a destroyed checkpoint? Right now the answer is to wait, and waiting is expensive. Any in-flight signal that predicted the branch out of the stability gap would change how aggressively teams are willing to re-warm, which is currently the binding constraint on how much a continued run can accomplish.

Where does the crossover between upcycling and from-scratch MoE actually sit? It is known to depend on the dense checkpoint's quality and on the MoE training budget, and it is known to move with both. It has not been characterised as a function anyone can evaluate before committing, which means the decision is still made by intuition and sunk cost.

Does mid-training need external scrutiny that pretraining data already gets? The stage is late, cheap and benchmark-sensitive, and it sits downstream of most published data documentation. That combination is a structural risk to benchmark interpretability regardless of anybody's intent, and no evaluation norm currently addresses it.

Sources and Further Reading

1. Ibrahim, A., Thérien, B., Gupta, K., Richter, M. L., Anthony, Q., Lesort, T., Belilovsky, E., & Rish, I. (2024). "Simple and Scalable Strategies to Continually Pre-train Large Language Models." Transactions on Machine Learning Research. arXiv:2403.08763
2. Gupta, K., Thérien, B., Ibrahim, A., Richter, M. L., Anthony, Q., Belilovsky, E., Rish, I., & Lesort, T. (2023). "Continual Pre-Training of Large Language Models: How to (re)warm your model?" arXiv:2308.04014
3. Abbes, I., Subbaraj, G., Riemer, M., Islah, N., Thérien, B., Tabaru, T., Kingetsu, H., Chandar, S., & Rish, I. (2025). "Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models." arXiv:2508.01908
4. Guo, Y., Fu, J., Zhang, H., Zhao, D., & Shen, Y. (2024). "Efficient Continual Pre-training by Mitigating the Stability Gap." arXiv:2406.14833
5. Grattafiori, A., et al. (2024). "The Llama 3 Herd of Models." arXiv:2407.21783
6. OLMo Team; Walsh, P., Soldaini, L., Groeneveld, D., et al. (2025). "2 OLMo 2 Furious." arXiv:2501.00656
7. Hu, S., et al. (2024). "MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies." COLM 2024. arXiv:2404.06395
8. Rozière, B., et al. (2023). "Code Llama: Open Foundation Models for Code." arXiv:2308.12950
9. Azerbayev, Z., Schoelkopf, H., Paster, K., Dos Santos, M., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., & Welleck, S. (2024). "Llemma: An Open Language Model For Mathematics." ICLR 2024. arXiv:2310.10631
10. Zhou, Y., Diao, J., Shang, J., Li, P., & Weng, R. (2026). "Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training." arXiv:2606.05610
11. Du, W., Luo, T., Qiu, Z., Huang, Z., Shen, Y., Cheng, R., Guo, Y., & Fu, J. (2024). "Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training." NeurIPS 2024. arXiv:2405.15319
12. Wang, P., Panda, R., Torroba Hennigen, L., Greengard, P., Karlinsky, L., Feris, R., Cox, D. D., Wang, Z., & Kim, Y. (2023). "Learning to Grow Pretrained Models for Efficient Transformer Training." ICLR 2023. arXiv:2303.00980
13. Chen, T., Goodfellow, I., & Shlens, J. (2016). "Net2Net: Accelerating Learning via Knowledge Transfer." ICLR 2016. arXiv:1511.05641
14. Kim, D., Park, C., Kim, S., Lee, W., Song, W., Kim, Y., et al. (2024). "SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling." arXiv:2312.15166
15. Komatsuzaki, A., Puigcerver, J., Lee-Thorp, J., Riquelme Ruiz, C., Mustafa, B., Ainslie, J., Tay, Y., Dehghani, M., & Houlsby, N. (2023). "Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints." ICLR 2023. arXiv:2212.05055
16. He, E., Khattar, A., Prenger, R., Korthikanti, V., Yan, Z., Liu, T., Fan, S., Aithal, A., Shoeybi, M., & Catanzaro, B. (2024). "Upcycling Large Language Models into Mixture of Experts." arXiv:2410.07524
17. Wei, T., Zhu, B., Zhao, L., Cheng, C., Li, B., Lü, W., et al. (2024). "Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models." arXiv:2406.06563
18. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., et al. (2017). "Overcoming catastrophic forgetting in neural networks." PNAS, 114(13), 3521-3526. PubMed 28292907
19. Dohare, S., Hernandez-Garcia, J. F., Lan, Q., et al. (2024). "Loss of plasticity in deep continual learning." Nature, 632, 768-774. doi:10.1038/s41586-024-07711-7

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.