Fault Tolerance for Long Training Runs

At a thousand accelerators over several weeks, hardware failure is not an exception to plan around; it is a scheduled event whose timing is unknown. Published accounts of frontier-scale training describe interruptions on the order of hours rather than weeks between failures, dominated by GPU faults, memory errors and network issues. A training system that treats failure as exceptional will spend more time restarting than training.

The checkpoint interval

Checkpointing trades a known cost against an expected loss. Writing more often costs more time in writes; writing less often loses more work per failure. With checkpoint duration \(C\) and mean time between failures \(M\), the interval minimising expected overhead is approximately

which is Young's classic result and remains the right first estimate. With a 60-second checkpoint and a 4-hour MTBF, that gives roughly 10 minutes. The important consequence is that reducing \(C\) lets you checkpoint more often and spend less total time doing it, so making checkpoints fast is worth more than tuning the interval.

Asynchronous checkpointing, where the state is copied to host memory quickly and written to storage in the background, and sharded checkpointing, where each rank writes only its own shard in parallel, are what bring \(C\) down from minutes to seconds at large scale.

What must be in the checkpoint

Model weights are the obvious part and the smallest problem. The optimiser state for Adam is two additional tensors per parameter, so it typically dominates. The learning rate schedule position, the data loader's position in the stream, and the random number generator states all matter for a resume to be equivalent to no interruption.

The data loader position is the one most often omitted. Resuming from the start of the dataset means recently seen examples are seen again immediately, which is a subtle corruption of the training distribution that produces no error and a slightly worse model.

Detection, which is the harder half

A crashed job is easy. The dangerous failures are the ones that do not crash.

A GPU with an uncorrectable memory error may produce wrong numbers rather than faulting. A degraded network link slows one rank without failing. A rank that hangs in a collective operation stalls the whole job with no error, and the default timeouts on collectives are long enough that hours can pass before anything is reported.

Practical systems therefore monitor per-rank step time and alert on divergence, watch for loss spikes and NaNs as a corruption signal, set aggressive collective timeouts so a hang surfaces in minutes, and run periodic health checks on hardware. Detecting a bad node is often harder than replacing it.

When it breaks

Restart is not free even when it works. Reacquiring capacity, reloading a large checkpoint, and re-warming caches takes time proportional to model size. At very large scale this becomes a substantial fraction of the interruption cost, and hot spares that already hold the weights are the response.

Checkpoint storage is a real capacity problem. Keeping every checkpoint of a large model consumes storage faster than most plans allow. A retention policy keeping recent checkpoints densely and older ones sparsely is necessary, and it must retain enough history to roll back past a corruption that was not detected immediately.

Rolling back requires knowing when the corruption started. If a bad node produced wrong gradients for two hours before detection, resuming from the most recent checkpoint resumes from a poisoned state. This is why loss-curve monitoring is a fault-tolerance mechanism and not just an observability nicety.

Elastic recovery changes the run. Continuing with fewer workers changes the effective batch size, which changes the optimisation trajectory. It is usually preferable to waiting for replacement and it makes the run not quite the run that was configured, which should be recorded in the run metadata rather than discovered when results are compared.