Keeping the training run alive

The only two complete public training logbooks (Meta's OPT-175B, BigScience's BLOOM) record what actually stops a big run: roughly two machine deaths a day, stalls that no timeout catches, one slow GPU taxing the whole fleet, and recovery machinery that fails more expensively than the hardware it guards. This guide turns those logbooks, the operator's handbook, and the 2023-26 tooling wave (DLRover, torchft, NVIDIA's resiliency extension, Orbax emergency checkpointing) into a reference architecture, a checkpoint-cadence formula, a four-class failure catalogue and a goodput-focused build ladder an architect can apply to their own cluster.

What you get out of it

• Steady-state attrition is the baseline, not the exception: ~2 machines/day on OPT's 128 hosts, a GPU death per week on BLOOM's 384 GPUs, and ~90 restarts to complete 33 ideal days of training.
• Checkpoint cadence is arithmetic, not folklore: BLOOM wrote it down (2.3 TB, 40 s saves, 100-iteration interval, 0.37% overhead), and tiered host-memory checkpointing has since collapsed the save cost that made intervals long.
• A deadline on collectives is not a deadline on progress: BLOOM lost 18 hours to a stall during which NCCL never timed out; progress must be watched from outside the process.
• Fail-slow hides below benchmark noise: one GPU cost 5% of fleet throughput while deviating only 2.5% in its own benchmark; detection has to be fleet-relative and continuous.
• The recovery path is a second production system: gate and drill node replacement, restores and restart automation like deploys, because they run when the system is already degraded.

Scope

Why this, now. Fault-tolerance for training just became installable software: Meta's torchft and NVIDIA's resiliency extension shipped their first releases two days apart, and reading them against the 2021-22 logbooks that motivated them is newly possible and newly useful.

What it does not cover. Loss spikes and numerical divergence, silent data corruption in depth, serving reliability, and the paper record (Llama 3 herd, MegaScale, GEMINI, SuperBench), which the session's network could not reach and which is named as a gap rather than quoted.