Replay Ratios and Forgetting in Continued Pretraining

Train a model on German after training it on English and the English loss rises. This is catastrophic forgetting, and the classical literature attacked it with clever machinery: penalise movement in parameters that mattered to the old task, weighted by the diagonal of the Fisher information matrix (Kirkpatrick et al., 2017, Overcoming catastrophic forgetting in neural networks, PNAS 114:3521-3526). Elastic weight consolidation is elegant, well-motivated as a Laplace approximation to the previous-task posterior, and largely not what frontier labs use.

What they use is rehearsal: mix a fraction of the original pretraining distribution back into the new corpus and keep training. It is the least sophisticated option available and it works better than the alternatives at scale.

Why mixing beats penalising

EWC constrains parameters. Replay constrains behaviour. At a few million parameters and a handful of tasks, a diagonal Fisher penalty is a reasonable proxy for "do not break what you knew". At ten billion parameters trained on a distribution rather than a task, the diagonal approximation is poor, the Fisher is expensive to estimate, and the penalty has to be tuned per layer to avoid either freezing the model or doing nothing.

Replay has none of those problems. The old distribution is already on disk, and sampling from it costs exactly the forward and backward pass it displaces. The gradient it contributes is the true gradient of the old objective rather than a curvature-based stand-in for it. The cost is also transparent: at replay fraction \(r\), a fraction \(r\) of your continued-pretraining compute is spent on data the model has already seen.

How much replay

Ibrahim et al. found that 5% replay, combined with learning-rate re-warming and re-decaying, was enough to match a full retraining baseline on both the old and the new distribution, including under an English-to-German shift (Ibrahim et al., 2024, arXiv:2403.08763). That number, 5%, has become a default in a lot of recipes, which is worth being slightly suspicious of, because it was measured at a particular scale and a particular shift strength.

More recent work complicates it usefully. Studying continual pretraining across languages with 100B tokens per language on Llama-family architectures, Abbes and colleagues report a scaling comparison: a 25% replay rate is a more compute-efficient use of budget than spending the same compute on a larger model, while a 50% replay rate is not (Abbes et al., 2025, Revisiting Replay and Gradient Alignment for Continual Pre-Training of LLMs, arXiv:2508.01908). The practical reading is that replay has a broad, forgiving optimum somewhere in the 5% to 25% band, and that the exact figure depends on how far the new distribution sits from the old.

A useful framing: replay is not free anti-forgetting, it is a purchase. You are buying retention of the old distribution with tokens that could have bought adaptation to the new one. Choose the fraction by deciding how much upstream degradation is acceptable, not by copying a number.

Gradient alignment as the middle ground

Between naive rehearsal and parameter penalties sits a third family that asks whether the replay gradient and the new-data gradient point in compatible directions, and corrects when they do not. Meta-experience replay approximates this. The 2025 study above gives an implementation with negligible compute and memory overhead over plain replay and reports more stable learning without forgetting. It is the one refinement of rehearsal with a clear empirical case at pretraining scale.

Plasticity, not just memory

Forgetting is the failure everyone anticipates. The opposite failure is quieter: a network trained continually can lose the ability to learn new things at all. Dohare et al. showed standard deep learning methods degrade toward the performance of a shallow network over long task sequences, with ImageNet binary-classification accuracy falling from 89% on an early task to about 77% by the 2,000th, roughly what a linear model achieves. Their mitigation, continual backpropagation, reinitialises a small fraction of the least-used units after each example, and L2 regularisation combined with weight perturbation also helped substantially (Dohare et al., 2024, Loss of plasticity in deep continual learning, Nature 632:768-774).

That result is from a many-task continual setting rather than a two-stage LLM pipeline, so do not transfer it uncritically. It does explain why teams running a model through five or six successive continued-pretraining rounds start to see diminishing gains per round that replay alone does not fix.

When it breaks

Replay assumes you still have the original corpus. Licensing changes, data deletion requests and simple infrastructure rot mean this is often false for a checkpoint acquired from someone else, which is the common case for open-weights models. The usual substitute is a proxy corpus drawn from a similar distribution, and it is genuinely weaker: you are replaying something adjacent to what the model learned, not what it learned.

Replay also cannot reverse a shift that is too strong for the token budget. If the new domain requires the model to overwrite representations rather than extend them, a small replay fraction will preserve surface fluency on the old distribution while the underlying capability still degrades, which is the worst outcome because the metric you are watching does not move.