Continued Pretraining and Learning-Rate Re-Warming

A 2 trillion token pretraining run finishes. Six months later there is a new corpus, or a domain the model is weak in, and the obvious move is to keep training. The obvious move is also the one that fails in a specific and instructive way: the checkpoint you are resuming from sits at the end of a decayed learning-rate schedule, where the optimiser has stopped moving. Feed it new data at that terminal learning rate and it barely learns. Feed it new data at the original peak learning rate and it forgets what it knew.

Continued pretraining (CPT) is the set of choices that gets you out of that bind. It is now a standard stage in production pipelines rather than a research curiosity: Code Llama was built by continuing Llama 2 for 500B additional tokens of 85%-code data (Rozière et al., 2023, Code Llama, arXiv:2308.12950), and Llemma was built by continuing Code Llama on Proof-Pile-2 (Azerbayev et al., 2024, Llemma, ICLR, arXiv:2310.10631).

Why the terminal learning rate is the problem

Almost every pretraining recipe ends with the learning rate annealed toward zero. That final decay is not incidental; it is where a large fraction of the loss reduction happens, and it leaves the weights in a sharp, well-fitted basin. Resuming at \(\eta \approx 0\) means the gradient from the new distribution is multiplied by nothing. The model sees the new data and does not move.

So you have to raise the learning rate again. The empirical study of exactly this is Gupta et al., who continued models pretrained on the Pile (300B tokens) onto SlimPajama (297B tokens) under different warm-up strategies. Their finding is counter-intuitive and load-bearing: re-warming initially increases loss on both the old and the new distribution, and that transient is the price of admission. Over a long enough continued run, the re-warmed model beats both the un-re-warmed one and a model trained from scratch on the union (Gupta et al., 2023, How to (re)warm your model?, arXiv:2308.04014).

The three-part recipe

Ibrahim et al. reduced the practical question to three knobs and showed the combination is sufficient: re-warm the learning rate, re-decay it over the continued run, and replay a small fraction of the original corpus. With those three in place, continued pretraining matched a full retraining baseline on final loss and on downstream evaluations, both for a weak shift (English to English) and a strong one (English to German), at 405M parameters and again at 10B (Ibrahim et al., 2024, Simple and Scalable Strategies to Continually Pre-train LLMs, TMLR, arXiv:2403.08763).

Re-decaying matters as much as re-warming. A continued run held at a constant elevated learning rate never gets the sharpening that the final anneal provides, and its evaluation numbers reflect that even when its training loss looks reasonable. Treat the continued run as a complete schedule in miniature: warm up, hold, decay to zero.

How high to re-warm is the part with the least settled answer. Common practice is somewhere between 10% and 50% of the original peak, scaled down as the distribution shift gets weaker and as the continued token budget gets smaller. Recent work argues the choice is not arbitrary at all: optimal continued-pretraining hyperparameters follow predictable scaling laws, and you can estimate a checkpoint's "equivalent pretraining compute" from its validation loss and then read the learning rate off a fitted curve, cutting hyperparameter search overhead substantially (Zhou et al., 2026, Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training, arXiv:2606.05610).

Schedules built for resumption

The cleanest fix is to not decay to zero in the first place. The warmup-stable-decay (WSD) scheduler warms up, then holds a constant peak learning rate for the bulk of training, then decays sharply over a short final window. Because the stable phase does not encode a total step count, any checkpoint taken during it is a legitimate resumption point, and you can branch a short decay off it whenever you want a deployable model (Hu et al., 2024, MiniCPM, COLM, arXiv:2404.06395).

That property is why WSD-style schedules spread quickly. A cosine schedule commits you to a horizon on step one. A WSD schedule leaves the run open, which is the right default once you accept that the training run is not going to end.

When it breaks

Continued pretraining does not rescue a bad base model, and it does not add capability that the architecture cannot represent. The failures that do occur cluster in three places.

The first is forgetting under a strong shift. If the new corpus is far from the original and there is no replay, the upstream loss climbs and general benchmarks fall, sometimes sharply. Replay is the mitigation, and it is covered separately.

The second is the transient dip. Loss gets worse before it gets better, and a team that evaluates 2,000 steps into a CPT run and sees degradation will often kill it. That dip has a name and a shape; see the stability gap.

The third is budget mismatch. Re-warming to a high learning rate and then stopping after a few billion tokens gives you the disruption without the recovery. If the continued budget is small (well under 1% of original pretraining tokens), re-warm gently or do not re-warm at all, and consider whether what you actually want is fine-tuning rather than continued pretraining.