The Stability Gap in Domain Adaptation

You start continued pretraining a general model on medical text. Two thousand steps in, medical benchmark accuracy has gone down. The intuitive reading is that something is misconfigured. The correct reading, usually, is that you are inside the stability gap: performance on the target domain dips at the beginning of continued pretraining and then recovers and surpasses the starting point, a pattern first characterised in continual learning for vision and then shown to apply to LLM domain adaptation (Guo et al., 2024, Efficient Continual Pre-training by Mitigating the Stability Gap, arXiv:2406.14833).

The gap matters for a practical reason. If your CPT run is a single epoch over a very large domain corpus, a large share of the budget is spent climbing back out of the dip rather than gaining ground beyond it.

Why the dip happens

The re-warmed learning rate disrupts the existing solution before the new data has supplied enough gradient signal to build a better one. Two forces are in tension. The optimiser is now taking large steps, which degrades a finely tuned set of weights immediately. The new distribution's contribution accumulates over many steps, which means the benefit arrives with a lag. Early in the run, disruption dominates. Later, accumulation does.

That framing predicts the shape correctly: the dip is deeper when the learning rate is re-warmed higher, deeper when the distribution shift is stronger, and shallower when the new corpus overlaps the old. It also predicts that the dip is not a bug to be eliminated so much as a transient to be shortened, because the same large steps that cause it are what make adaptation possible at all.

Three strategies that shorten it

Guo et al. propose and test three interventions, and the first is the one most teams get wrong by default.

Multiple epochs over a smaller subset, instead of one epoch over everything. Given a fixed token budget, repeating a well-chosen subset recovers performance faster than a single pass over the full corpus. This inverts the usual pretraining instinct, where repetition is treated as something to avoid because of memorisation risk. At CPT scale with a modest budget, the faster recovery is worth more than the marginal diversity.

Pretrain on the high-quality sub-corpus only. Filtering the domain corpus hard and training on the top slice boosts domain performance quickly, which matters precisely because the early steps are where the dip lives.

Keep the mixture close to the original pretraining distribution. Reducing the distribution gap directly reduces the size of the disruption. This is the same lever as replay, seen from the other side.

Applied together on OpenLlama-3B, these raised average medical task performance from 36.2% to 40.7% using 40% of the original training budget, while general task performance improved rather than degraded (Guo et al., 2024, arXiv:2406.14833).

Reading the curve correctly

The operational lesson is about instrumentation. Evaluate the target domain and the general capabilities on the same cadence from step zero, and expect both to move. A CPT run where the target metric climbs monotonically from the first evaluation usually means the learning rate is too low to be doing much, and the run will underperform a more aggressive one that dips first.

Set the kill criterion on the shape rather than on a single reading: a run that has not begun recovering by roughly the point where its cumulative token count reaches a few percent of the original pretraining budget is genuinely in trouble; one that is still in the dip at 2,000 steps is not.

When it breaks

The recovery is not guaranteed. If the re-warmed learning rate is high enough to push the model out of its basin entirely, there is no recovery phase, only a slow re-descent from a much worse starting point, and the run is strictly worse than starting from a base checkpoint with a gentler schedule. Distinguishing "deep dip that will recover" from "destroyed the checkpoint" during the run is genuinely hard, which is the main argument for conservative re-warming when the token budget is small.

The gap also interacts badly with early stopping on validation loss. Upstream validation loss during a strong-shift CPT run can rise for a long stretch while the target capability is improving. A stopping rule that watches only the old distribution will terminate every successful run.