Fault Tolerance at Scale

Meta's Llama 3 405B pre-training reported 466 job interruptions over a 54-day snapshot on a 16,384-GPU H100 cluster, of which 419 were unexpected, with GPU failures accounting for roughly 30 percent and HBM3 memory failures a further 17 percent (Llama 3 team, 2024, The Llama 3 Herd of Models, arXiv:2407.21783). That is a failure roughly every three hours. At that rate, fault tolerance is not an operational nicety bolted on at the end; it is a property the training system has to be designed around.

Why failure rate scales with device count

Synchronous data-parallel training is a lockstep computation: every device must complete every step. A single failed device halts the entire job. If each device has an independent failure probability, the job's mean time between failures falls roughly inversely with device count.

A device that fails once a year is unremarkable. Sixteen thousand of them produce a failure every few hours, and the arithmetic is why the same hardware reliability that is invisible at 8 GPUs becomes the dominant engineering concern at 10,000.

Checkpoint interval is an optimisation problem

Two costs pull against each other. Checkpointing too rarely wastes the work done since the last save; checkpointing too often spends the run's time writing. With mean time between failures \(T_f\), checkpoint write cost \(C\), and interval \(I\), expected overhead per unit of useful work is approximately

The first term is the write tax, the second the expected lost work (on average half an interval). Minimising gives the classic result

Plug in real numbers and the tension is clear. A checkpoint of a 400B-parameter model with optimiser state is several terabytes; writing it synchronously to shared storage can take minutes, during which every GPU is idle. With \(T_f\) measured in hours, the optimal interval lands in the tens of minutes, and the write tax alone is a nontrivial fraction of the run.

What made this tractable

Sharded checkpointing. Each rank writes only its own shard in parallel rather than gathering to rank 0, turning an \(O(\text{model size})\) serial write into an \(O(\text{shard size})\) parallel one.

Asynchronous checkpointing. Copy the state to host memory quickly, then let a background thread write to durable storage while training continues. GPU idle time falls to the copy, not the write.

In-memory and peer checkpointing. Keep a recent checkpoint replicated in the DRAM of other nodes, so a single-node failure restores from a neighbour in seconds rather than from object storage in minutes. Durable storage then only needs the slower, less frequent copies.

Elastic and hot-spare scheduling. Keep spare nodes in the cluster so a failed node is replaced without waiting for the scheduler, and re-form the process group rather than restarting the job from the shell.

When it breaks

• The checkpoint is tied to the parallel layout. Restarting on a different mesh requires resharding, which mature frameworks support and hand-rolled trainers usually get subtly wrong. Test the restart path before you need it.
• Data loader state is part of the checkpoint. Restoring weights but not the data iterator silently re-trains on the same tokens and skips others, which shows as a mysterious quality regression rather than an error.
• Stragglers are worse than failures. A device that fails is detected and replaced. A device that runs 30 percent slow, from thermal throttling or a degraded link, slows every step for the entire job and reports no error at all. Per-rank step-time monitoring is the only way to find it.
• Silent data corruption. A small fraction of hardware faults corrupt values without crashing anything, producing NaNs much later or, worse, plausible but wrong gradients. Loss-spike detection and periodic determinism checks are the practical defences. See loss spikes and divergence.
• Failures cluster. They are not independent: a rack power event or a network partition takes many nodes at once, so the MTBF estimate above is optimistic in exactly the situations that hurt most.