A frontier training run is one synchronous computation spread across thousands of machines, where a fault in any single part stalls every other part, and the parts fail daily. This guide reconstructs how teams survive that: from the two complete public training logbooks (Meta's OPT-175B and BigScience's BLOOM), the handbook their operator wrote afterwards, and the fault-tolerance code the industry shipped since. Afterwards you can set a checkpoint cadence from arithmetic rather than fear, and say which failure class your watchdogs will miss.
The problem, stated without naming a framework: one computation must run for weeks across tens of thousands of parts, any part can stall the whole, and progress survives only through state that was deliberately saved.
Training a large model is the one modern workload with mainframe failure semantics. A web fleet loses a machine and sheds a sliver of capacity; a synchronous training job loses a machine and produces nothing at all until that machine's work is re-homed, because every rank waits on every other rank at every step. Meta's OPT-175B logbook records what that means: "~2 machines go down every day" out of 128 hosts, and roughly 90 restarts to complete what would have been 33 uninterrupted days of compute. BigScience's BLOOM chronicles show the same shape at 384 GPUs: a GPU death "every week or so", absorbed by automation, and occasional stalls that were not. The stakes have grown since: Meta reports the Llama 3.1 family consumed 39.3 million H100-hours of training compute, which prices an hour of cluster downtime in thousands of GPU-hours.
The public record of this problem has an unusual shape, and this guide leans into it. The two most detailed accounts ever published are engineering logbooks written during the incidents, in 2021 and 2022, by teams who deliberately shipped their notes with the model. Nothing as complete has appeared since, even as clusters grew tenfold; the quantitative accounts of the 10,000-GPU era live in papers this session's network could not reach (section 5 names them). What became public instead is code: between January 2023 and October 2024, Ant Group, Meta and NVIDIA each shipped an open-source fault-tolerance layer, and the PyPI record dates Meta's torchft (2024-10-13) and NVIDIA's resiliency extension (2024-10-15) two days apart. The lessons of the logbooks took about three years to become installable software, and reading the two eras against each other is the most instructive thing this corpus offers.
The most expensive incidents in both logbooks were not GPUs failing; they were the recovery machinery failing. OPT's worst single incident was the cloud provider's support team accidentally deleting the entire cluster while replenishing the spare-node pool; its checkpoint restores hung on blob storage; BLOOM's crash-to-restart chain routinely failed to fire; and PyTorch's own diagnostic dump can fail under exactly the error-handling setting operators are told to enable. Treat detection, checkpointing, spares and restart as a second production system with its own failure modes, because that is what the record shows it to be.
Scope: this guide covers keeping a synchronous training job running against hardware and infrastructure failure: detection, checkpointing, spare capacity, restart, and per-step recovery. It deliberately excludes loss spikes and numerical divergence (a training-science problem the same logbooks also document), silent data corruption in depth, and serving reliability. It also, by necessity, excludes the paper record: this session could reach only GitHub and the package registries, which is why every citation is a logbook, an issue thread, source code or a registry timestamp, and why the famous paper numbers are named as a gap rather than quoted.
Five organs recur across every system in the corpus: screening before the run, layered detection during it, tiered state preservation, a spare-capacity pool, and restart orchestration. The divergence point is what restarts: the whole world, or one replica group.
Screening exists because the delivered fleet is not the working fleet. The OPT team found that replacing a node through the cloud interface "can take hours for a single machine" and that "more often than not we would end up getting the same bad machine again", so they built their own GPU burn-in and InfiniBand tests and scripted the replacement. Bekman's fault-tolerance handbook, distilled from operating BLOOM, puts a number on why: "There can be as large as 10% failure rate early on for new accelerators", and one failed accelerator idles the other eight on its node. NVIDIA's resiliency extension now ships "system-wide health checks" as a product feature, which is the 2021 shell script promoted to a supported layer.
Detection is layered because each layer fails. Inside the process, PyTorch's NCCL
watchdog enforces a
10-minute
default timeout per collective (kProcessGroupNCCLDefaultTimeout). Watching
the watchdog, a heartbeat monitor tears the process down if the watchdog thread itself stops
making progress for
8
minutes, because, as the header comment explains, CUDA and NCCL calls "may hang" and the
alternative is "jobs being stuck for a prolonged time than necessary tying up cluster
resources". Outside the process sits the layer the 2024 poster describes: a flight-recorder
ring buffer of recent collectives, dumped on timeout, and a per-node HTTP
WorkerServer
that an independent monitoring service polls for dumps, NCCL state and py-spy profiles. That
third layer is exactly the tool the BLOOM operator lacked in April 2022, when the job stalled
with "gpus spinning at 100%" and diagnosis meant hand-running py-spy over SSH; the mechanism
was institutionalised rather than invented.
State preservation is a tier ladder, not a file. The 2022 systems wrote one artefact to one place: BLOOM saved 2.3 TB to a parallel filesystem every 100 iterations, OPT pushed 992 files to blob storage. The current systems copy device state synchronously to host shared memory in seconds, persist asynchronously from there, and restore from the fastest tier that survived the failure: DLRover's flash checkpoint reloads "directly from the host memory" when only the process died, and Google's Orbax emergency checkpointing defaults to a local save every 10 steps against a much rarer persistent one. Figure 3 draws the ladder; the decision section prices it.
Spare capacity and orchestration close the loop. OPT ran a buffer pool that grew from 4 hosts to 18, plus 12 for the holidays, as correlated failures overwhelmed the smaller pool, and by the end the pipeline of health checks, node swap and resume had recovered from 8 failures unattended in about a week. The divergence point among current systems is the restart unit. The stop-the-world school (DLRover, NVIDIA's in-job restart) keeps the allocation but restarts the processes in place, skipping the scheduler queue. The per-step school (torchft, wired into torchtitan) makes the replica group the failure domain: a quorum service admits and evicts groups "at the training step granularity", and a healed group receives weights by "live recovery from a healthy peer" rather than from storage. Bekman states the prize plainly: with replication "you only ever lose one iteration, whereas with file system checkpointing you may lose hundreds of iterations". The cost is that it requires data-parallel replication and a redesigned training loop, which is why it shipped as a library with a coordinator (the lighthouse) rather than as a flag.
Collective timeout (10 min default), watchdog heartbeat (8 min), flight-recorder dump, external poller with py-spy. Each layer exists because the one below it can hang.
Seen at: PyTorch c10d, Meta poster, NVIDIA NVRx
Tiered saves: device to host shared memory synchronously, then async to local and remote. Restore prefers the fastest surviving tier. Keep two remote checkpoints; the newest can be torn by the crash that ended the run.
Seen at: DLRover flash, Orbax emergency, Bekman
Validated buffer nodes sized for correlated failure, scheduler-level teardown so a dead rank cannot leave the job in limbo, and either in-place restart or per-step quorum over replica groups.
Seen at: OPT logbook, DLRover design, torchtitan + torchft
Each decision below was faced, in writing, by at least one team in the corpus. The flips-when column is the rule you can reuse.
NCCL_ASYNC_ERROR_HANDLING=1, "force crashing on nccl issues like hanging broadcast",
plus scheduler-level teardown, because a stall costs unbounded time and a crash costs
one interval.abort(), decides. Fail-fast is correct
until the day you install something that can do better, and wrong after it.| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Spare capacity | Own validated buffer pool (4, then 18+12 hosts) | Replace via provider on demand | Replacement took hours and "more often than not" returned the same bad machine | OPT logbook, 2021 |
| Node admission | Burn-in and fabric tests before joining | Trust the scheduler's view of health | ~10% early-life accelerator failure; nodes arrive broken in ways schedulers cannot see | Bekman, NVRx |
| Who tears down a stuck job | The scheduler (--kill-on-bad-exit), outside the process | The training runtime handles its own exit | torch.elastic left workers "hanging in an infinite state of logging for 8 hours" | pytorch #76287, 2022 |
| Restart placement | In-job restart on standby workers | Requeue through the scheduler | Reallocation waits in queue; NVRx restarts "without the need to reallocate SLURM nodes" | NVRx README, DLRover design |
| Checkpoint destination | Tiered: host memory, local, then remote | Remote store only | Restores from remote hung (OPT); host-memory reload recovers a process crash in seconds | OPT logbook, DLRover flash |
| Operator override | Stop-file polled by the training loop | Rely on scheduler permissions | On the shared HPC machine nobody else could stop a colleague's job; the file could be created by anyone on the team | BLOOM chronicles |
The logged incidents sort into four classes: hard faults, silent stalls, fail-slow, and failures of the recovery machinery itself. The fourth class is the expensive one, and the one designs least often account for.
"For some reason NCCL wasn't timing out either! even after 3 hours of not being able to broadcast." BigScience BLOOM training chronicles, entry of 2022-04-30
CUDA error: unknown error during a collective; the job died 2 iterations before the scheduled save.NCCL_ASYNC_ERROR_HANDLING=1 plus srun --wait=60 --kill-on-bad-exit=1: make communication errors fatal and let the scheduler end the job.TORCH_NCCL_ASYNC_ERROR_HANDLING=1, per an issue a PyTorch maintainer filed and later closed as not planned.--kill-on-bad-exit, --wait=60, plus an external watchdog on log progress.Everything quantitative the corpus yields, dated and sourced. Measured means a primary logbook or a code constant; claimed means a project's own README; derived shows its arithmetic in the ledger.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Machine attrition | ~2 hosts/day | Meta OPT | 128-host A100 cluster, steady state (measured) | 2022-01 | logbook |
| Restarts, whole run | ~90 | Meta OPT | Against ~33 ideal days on 1,024 A100s; ~1.6/day derived | 2022-01 | logbook |
| Restarts, worst fortnight | 40+ | Meta OPT | Hardware-dominated stretch around Thanksgiving (measured) | 2021-12 | logbook |
| Record uninterrupted runs | 1.5 / 2.8 / 2 days | Meta OPT | Best stretches, a month into the run (measured) | 2021-12 | logbook |
| GPU attrition | ~1 death/week | BigScience BLOOM | 384 A100s on Jean Zay; auto-restart absorbed most (measured) | 2022-04 | chronicles |
| Checkpoint size / save time | 2.3 TB / 40 s | BigScience BLOOM | Written concurrently from 384 processes to GPFS over NVMe (measured) | 2022-03 | chronicles |
| Checkpoint overhead, whole run | 0.37% | BigScience BLOOM | ~720 saves over ~3 months; 2% if IO were 5x slower (measured/derived) | 2022 | Bekman |
| Worst single losses | 7.3 h / 18 h / 11 h | BigScience BLOOM | Crash just before a save; two undetected stalls (measured) | 2022 | chronicles |
| Fail-slow cost | 5% throughput | BigScience BLOOM | 149 to 140 TFLOPs from one slow GPU; its own benchmark deviated 2.5% (measured) | 2022-04 | chronicles |
| Collective timeout default | 10 min | PyTorch NCCL | kProcessGroupNCCLDefaultTimeout (code constant) | 2026-09 | c10d source |
| Watchdog heartbeat deadline | 8 min | PyTorch NCCL | Monitor aborts the process if the watchdog itself hangs (code constant) | 2026-09 | c10d source |
| Goodput delta, fault tolerance | 69% → 95% | Ant Group | GLM-65B on thousands of GPUs (claimed by the operator) | 2023-08 | DLRover |
| Goodput delta, flash checkpoint | 90% → 95% | Ant Group | Persistence time cut ~70x via a shared-memory tier (claimed) | 2024-01 | DLRover flash |
| Local checkpoint cadence | every 10 steps | Google Orbax | Default for the emergency local tier (code default) | 2026-09 | source |
| Training compute at stake | 39.3M GPU-h | Meta Llama 3.1 | Cumulative H100-hours for the model family (vendor figure) | 2024-07 | model card |
The DLRover goodput figures are the operator's own claims about its own product, on workloads it chose; this corpus holds no independent measurement of them. The 2022 logbook numbers are measured but predate today's cluster sizes by an order of magnitude: failure rates scale with part count, so treat the OPT and BLOOM attrition figures as per-node baselines rather than per-cluster ones. The widely quoted interruption statistics for the 16,384-GPU era come from the Llama 3 herd paper and from the MegaScale, GEMINI and SuperBench papers; none was reachable from this session, so none is quoted here. Fetch them yourself before planning against second-hand retellings of their numbers.
Every source behind this page, graded. The full ledger, one copied quote per
claim, ships beside this file as sources.md.
This session's network reached only GitHub and the package registries, so the wall holds no papers and no engineering blogs hosted elsewhere; that gap is stated in section 5 rather than papered over. The compensation: the two best artefacts on this topic are GitHub markdown written by the on-call engineers during the incidents, and they are graded postmortem because that is what they are.
The hardware fortnight: 40+ restarts, hours-long node replacement that returned the same bad machine, 1.6 TB checkpoint downloads hanging on blob storage, and the birth of their burn-in and InfiniBand test tooling.
Record uninterrupted runs of 1.5 to 2.8 days a month in; five hosts down together against a buffer of four; the buffer raised to 18; a volunteer on-call rotation with runbooks.
~90 restarts against 33 ideal days; ~2 machines lost daily; the provider's support team deleting the whole cluster while replenishing spares; then 8 unattended recoveries over the holidays and a 14-days-without-humans goal. The full 148-page logbook PDF sits beside it in the repo.
The moment the failure mix flips: once training settings stabilised, "the only restarts we've had to make were all related to hardware issues (missing GPUs on instances, training randomly hanging after including a new node, ECC errors, partial checkpoint upload after hardware error, CUDA errors, NCCL errors, etc.)."
A GPU crash 2 iterations before checkpoint time costs 7.3 hours; the interval is cut to 100 iterations with the arithmetic written out: 2.3 TB, 40-second saves, at most 3 hours of exposure.
A GPU dies "every week or so"; auto-restart usually absorbs it at a max-3-hour cost. Stalls defeat the automation, and shared-HPC permissions forced the team to invent a stop-file so anyone on the team could end anyone's job gracefully, checkpoint included.
A 5% fleet-wide throughput drop, clean network tests, and a weekend of excluding nodes three at a time to binary search the culprit, whose own benchmark deviated just 2.5%.
The eval-time deadlock no timeout caught, diagnosed by hand with py-spy; then the
2022-05-18 entry where NCCL_ASYNC_ERROR_HANDLING=1 and scheduler-level
teardown became the standing configuration.
Filed by BLOOM's operator from the incident above: the elastic launcher left workers of the failed node hanging for 8 hours. Still open as of 2026-09, four years on.
kProcessGroupNCCLDefaultTimeout is 10 minutes per collective;
TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC defaults to 8 minutes on the watchdog
itself, after which the monitor thread aborts the process. The header comments state the
reason: CUDA/NCCL calls "may hang", and stuck jobs tie up cluster resources.
A maintainer's own report that "calling commAbort from inside watchdog thread leads to exit, 'broken promise' exception from async dump thread, and no dump file". Closed as not planned in 2026.
A quorum service (the lighthouse) plus per-replica-group managers allow "membership changes at the training step granularity", with "live recovery from a healthy peer" instead of a storage restore, and process groups that "report errors sanely and be reinitialized gracefully".
The end-to-end integration: replica groups as separate torchtitan instances, tolerance of one group's loss, and the operational detail that "the alive replica group with the smallest replica ID will perform checkpointing saving".
DataStates-LLM, an asynchronous checkpointing engine with non-blocking GPU-to-host copies, offered as a contribution in July 2024 and closed four months later: "I'm going to close this PR now as stale, but we would appreciate the contribution if you are able to come back to this and resolve the merge conflicts?"
A two-tier checkpoint manager for multi-slice TPU jobs: a local tier defaulting to a save every 10 steps, a persistent tier saved rarely, and restore logic that prefers whichever local copy survived.
The detection layer in one page: flight recorder classifying deadlocks, stragglers, mismatched collectives and timeouts; a per-node HTTP WorkerServer for dumps, py-spy and fault injection; live checkpoint recovery over HTTP from healthy workers; fault-tolerant HSDP continuing "on the next batch without downtime".
The BLOOM operator's distilled handbook: plan more nodes than needed, expect ~10% early-life accelerator failure, the checkpoint-overhead arithmetic (0.37% for BLOOM, 2% with 5x slower IO), watchdog patterns, and the stop-file and save-file mechanisms.
The operator's account of running fault-tolerance automation across its Kubernetes fleet: automatic diagnosis, process-level versus node-level restart, and the measured claim that GLM-65B goodput rose from 69% to 95% on thousands of GPUs.
Synchronous copy to host shared memory, asynchronous persistence, emergency persist on failure, and memory-direct reload on process restart; wasted checkpoint time cut ~5x, persistence ~70x, goodput from 90% to 95%.
DLRover's failover mechanisms written up as a design document with per-scenario paths (process failure, node failure, master failure), rather than as operational folklore.
Hung-rank detection, in-job restart without reallocating scheduler nodes, async and local checkpointing, straggler detection, health checks; integrated into NeMo and Megatron-based stacks.
The family consumed a cumulative 39.3M GPU-hours on H100-80GB hardware; the training fleet behind it makes each hour of downtime worth thousands of GPU-hours.
dlrover first published 2023-01-16; torchft 0.1.0 on 2024-10-13; nvidia-resiliency-ext 0.1.3 on 2024-10-15. Meta's and NVIDIA's fault-tolerance packages first shipped two days apart, three years after the logbooks that motivated them.
Six rungs from a resumable loop to a goodput dashboard. The line from toy to real is crossed at rung 4, where you start injecting the failures instead of waiting for them.
Single-node PyTorch training loop with atomic checkpoint-and-resume (write to a temp name, fsync, rename; keep the last two). End the process abruptly at random points, including mid-save.
Done when: a hundred abrupt exits at random moments never lose more than one interval and never leave a corrupt checkpoint. Teaches: why the newest checkpoint is not always a usable checkpoint.
Run it under torchrun with more than one worker, wrapped by a scheduler (SLURM job array, or a Kubernetes Job with restartPolicy) so a failed process brings the job back without you. Reproduce BLOOM's settings: errors fatal, scheduler-level teardown.
Done when: ending any single worker gets the job back to training unattended, within one interval. Teaches: the restart chain has more links than you think, and each one can hold the job in limbo.
Add an external watchdog: a process on another machine that alerts when the step counter stops advancing, regardless of process state. Enable the flight-recorder environment variables and capture a dump from a deliberately mismatched collective.
Done when: a simulated stall (a rank sleeping inside a collective) is detected in minutes by the watchdog while the scheduler still says RUNNING. Teaches: the difference between a deadline on collectives and a deadline on progress.
Add a shared-memory or local-disk tier (DLRover flash checkpoint, or torch.distributed async checkpointing) under the remote tier. Measure save cost per tier, then set each tier's interval from your measured failure rate and tolerable loss, BLOOM-style.
Done when: a process restart restores from memory in seconds while a simulated node loss falls back to the remote tier. Teaches: the cadence question dissolves once the save is cheap.
Script the four classes from section 4 against your miniature: hard fault (end a worker), stall (block in a collective), fail-slow (throttle one GPU's clocks or add sleep-per-step to one rank), and a recovery-machinery fault (make the checkpoint store unwritable mid-run). Record detection latency and loss for each.
Done when: you have a table like section 5's for your own system, including the injected recovery-path failure. Teaches: which class your stack is blind to; for most stacks it is fail-slow or the recovery path.
Run torchtitan with torchft on one machine: two replica groups of four GPUs (or scaled down), end one group mid-run, watch quorum shrink and heal. Then build the dashboard that turns all of it into one number: goodput, as useful-step time over elapsed time, with downtime attributed to detection, restore and recompute.
Done when: ending a replica group costs seconds of the survivors' time, and your dashboard attributes every lost GPU-hour to a class. Teaches: goodput is the only metric leadership needs from this whole topic, and you can now defend its components.
The queries that actually found this material. The logbook genre hides in repository files, not on blogs, so several of these search file paths rather than the web.
site:github.com chronicles.md training logbook"we lost" hours training checkpoint site:github.comOPT logbook metaseq chronicles restarts"buffer nodes" OR "spare nodes" GPU training failuresrepo:pytorch/pytorch "flight recorder" in:titleTORCH_NCCL_HEARTBEAT_TIMEOUT_SEC defaultNCCL_ASYNC_ERROR_HANDLING hang slurmis:pr is:closed is:unmerged checkpoint async (in a framework repo)"goodput" GPU training cluster fault tolerance"in-job restart" OR "flash checkpoint" OR "emergency checkpoint"torchft lighthouse quorum replica grouppypi release history nvidia-resiliency-ext torchft dlroverLlama 3 herd paper unexpected interruptions tableMegaScale NSDI 2024 stragglers diagnosisGEMINI SOSP 2023 in-memory checkpointsSuperBench ATC 2024 gray failure proactive validation