Loss Spikes and Divergence

Plot the loss curve of nearly any large, long pretraining run and, sooner or later, the smooth downward line breaks: a sudden jump upward, sometimes recovering within a few hundred steps, sometimes never recovering without intervention. Chowdhery et al.'s PaLM paper is unusually candid about this, reporting roughly twenty such spikes across the full training run despite standard mitigations (warmup, gradient clipping, careful initialisation) being active the entire time. Loss spikes are not a sign of an amateur setup; they are close to a universal feature of training at this scale, and how a team responds to them is one of the least publicised but most consequential parts of running a frontier pretraining job.

What actually causes one

No single mechanism explains every spike, but three recur across published post-mortems.

Degenerate batches. A shard of highly repetitive, corrupted, or otherwise unusual data can produce a gradient that is unusually large and, worse, correlated across many parameters in the same direction. Global-norm gradient clipping rescales the whole vector's magnitude but preserves its direction, so a correlated bad gradient still pushes every affected parameter the same way, just by a smaller amount. Clipping blunts the damage; it does not neutralise a systematically bad update.

Adam's per-parameter scaling going stale. Adam divides each parameter's update by sqrt(v_hat), a running estimate of that parameter's typical squared gradient magnitude. If a parameter has had a quiet stretch (small v_hat) and then receives an unusually large gradient, its individual update can be large even after the gradient vector as a whole was clipped, because clipping applies a single global rescaling factor, not a per-parameter one. A handful of parameters going briefly out of sync with their own recent history is enough to move the whole model measurably.

Precision effects. Training in bf16 gives roughly three decimal digits of mantissa precision. If the softmax normaliser (the log-partition function inside cross-entropy, see softmax-logits) drifts to an unusually large or small value, small numerical errors in that computation can translate into a disproportionately large gradient. PaLM's mitigation was an auxiliary z-loss term that lightly penalises the log of the softmax normaliser for straying from zero, keeping the numerics in a well-behaved range without materially changing what the model learns. This interacts directly with format choice; see mixed-precision-bf16-fp8 for the broader precision picture.

What teams actually do

The pragmatic response documented in the PaLM report is unglamorous: roll the training state back to a checkpoint saved roughly a hundred steps before the spike, skip the batches of data that were being consumed around the time of the spike, and resume from there. This happened repeatedly over the course of the run. It is a workaround, not a fix, and it depends on checkpointing and data-shard bookkeeping being frequent and precise enough to pinpoint what to skip. OPT's public training logbook documents a similarly hands-on process of restarts, hyperparameter adjustments, and hardware failures interleaved with loss instability, offering one of the few fully transparent accounts of how messy this part of training actually is.

Prevention leans on everything else in this bundle

Spikes correlate with running close to the edge of what the current learning rate and batch size can tolerate: a learning rate set too high relative to the critical batch size, insufficient warmup, or an initialisation and width combination that has not been tuned to keep update sizes stable (the problem muP targets directly). None of these individually guarantees a spike-free run; together they shrink how often one occurs and how bad it is when it does.

When it falls down

• Spikes are not all alike. Some self-correct within a few hundred steps with no intervention; others compound and require an explicit rollback. Telling the two apart early, from the loss curve alone, is not reliable.
• The fix is still mostly reactive. Despite years of large-scale training experience across labs, there is no first-principles method to predict exactly when a spike will occur; monitoring and rollback remain the default response.
• Causes are codebase- and setup-specific. A fix that worked for one lab's spike (a particular z-loss coefficient, a particular clip threshold) does not reliably transfer to a different architecture, data mix, or precision setup.
• Aggressive monitoring has a real cost. Catching a spike early enough to intervene cheaply requires frequent checkpointing and per-layer gradient norm logging, both of which add overhead to every step of a run that, most of the time, does not need them.

Further reading

• Chowdhery et al., 2022, PaLM: Scaling Language Modeling with Pathways, arXiv:2204.02311 - documents loss spikes and the z-loss and rollback mitigations.
• Zhang et al., 2022, OPT: Open Pre-trained Transformer Language Models, arXiv:2205.01068 - the accompanying training logbook is a rare public record of instability and recovery in real time.