Evidence ledger
23 sources
Checked 26 Sep 2026
Evidence ledger
One row per claim in Keeping the training run alive: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
Topic: how teams keep a multi-week, multi-thousand-GPU training run alive on hardware that fails daily, reconstructed from the only two complete public training logbooks (Meta's OPT-175B, BigScience's BLOOM), the operator's own handbook, the fault-tolerance code written since (PyTorch's NCCL watchdog and flight recorder, torchft, NVIDIA's resiliency extension, DLRover, Orbax emergency checkpointing), and the issue and pull-request record around them.
All links fetched 2026-09-26. One row per claim. Quotes are copied, not paraphrased. (E) marks
an exact quotation, (R) a figure reported by the artefact, (D) a figure derived here from
reported figures, with the arithmetic shown.
Evidence limits, stated up front
- The network policy for this session blocked most of the open web.
arxiv.org,usenix.org,engineering.fb.com,imbue.com,pytorch.org,semianalysis.com,epoch.ai,huggingface.coandnvidia.github.ioall refused at the egress proxy. Only GitHub and the package registries were reachable. There are therefore no papers and no engineering-blog posts outside GitHub in this ledger. The famous quantitative accounts of training-run failure (the Llama 3 herd paper's interruption table, ByteDance's MegaScale paper, Meta's cluster-reliability paper, Amazon's GEMINI paper, Microsoft's SuperBench paper) could not be fetched and are therefore not cited; the guide names them in its gap note and does not use their numbers. - The compensation is unusually good for this topic. The two most valuable primary artefacts
in this field, the OPT-175B logbook and the BLOOM training chronicles, are markdown files in
GitHub repositories, written by the people on call, during the incidents. They are graded
postmortembecause that is what they are: dated incident entries with root cause, loss figures and remediation. - One talk is cited from its poster. The PyTorch Conference 2024 poster on fault tolerance ships as a PDF inside the torchft repository and was read in full; no video source was reachable, so claims are cited to the poster text rather than to timestamps.
- Host diversity is capped by the cage. Three hosts (github.com, raw.githubusercontent.com, pypi.org) carry the whole ledger. The distinct-organisation count is the honest breadth signal here: Meta (two eras: OPT 2021 and torchft 2024-26), BigScience/Hugging Face, the PyTorch project, NVIDIA, Microsoft (DeepSpeed), Ant Group (DLRover), and Google (Orbax).
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | Meta | OPT-175B chronicles, "Update on 175B Training Run: 27% through" | postmortem | 2021-12-03 | 2026-09-26 | https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/27_percent_update.md | Two and a half weeks of the run produced 40+ restarts, dominated by hardware | (E) "Since then, we've had 40+ restarts in the 175B experiment for a variety of hardware, infrastructure, or experimental stability issues." |
| 2 | Meta | Same document | postmortem | 2021-12-03 | 2026-09-26 | https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/27_percent_update.md | Provider replacement was slow and returned the same bad machine, forcing the team to build its own screening | (E) "Replacement through the cloud interface can take hours for a single machine, and we started finding that more often than not we would end up getting the same bad machine again." |
| 3 | Meta | Same document | postmortem | 2021-12-03 | 2026-09-26 | https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/27_percent_update.md | The restore path failed too: checkpoint downloads from blob storage hung nondeterministically | (E) "There were also issues with blob store when downloading 1.6TB of a single model checkpoint (992 files, each ~1.7GB) on restarts, at which point the downloads themselves would start hanging nondeterministically, which then delayed training recovery even further." |
| 4 | Meta | OPT-175B chronicles, "Update on 175B Training Run: 56% through" | postmortem | 2021-12-16 | 2026-09-26 | https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/56_percent_update.md | A month in, the record uninterrupted runs were 1.5 to 2.8 days; failures were correlated (5 hosts at once against a buffer of 4) | (E) "We managed to hit our top three record long runs of the experiment these past two weeks, lasting 1.5, 2.8, and 2 days each!" and (E) "especially considering 5 hosts going down all together on Dec 6 (back when we only had a buffer of 4 hosts)"; buffer then raised: (E) "Our cloud provider has also agreed to provision an extra 18 hosts for us to use as a 'buffer'". |
| 5 | Meta | OPT-175B chronicles, "175B Training Run Completed" | postmortem | 2022-01-07 | 2026-09-26 | https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/final_update.md | The full run took ~90 restarts against an ideal 33 days on 1024 A100s, with ~2 machines dying per day | (E) "roughly ~33 days of continuous training on 1024 80GB A100s (assuming no hardware issues, no numerical instabilities, etc.)", (E) "These notes cover ~90 restarts over the course of training the lineage of this current model", (E) "~2 machines go down every day". |
| 6 | Meta | Same document, "Cluster Deletion" section | postmortem | 2022-01-07 | 2026-09-26 | https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/final_update.md | The worst single incident was the recovery machinery itself: the provider's support team deleted the entire cluster while replenishing spare nodes | (E) "In the process of replenishing this pool, the cloud provider's support team accidentally deleted our entire cluster on December 21." and (E) "it unfortunately came back with 16 machines that did not pass our infrastructure checks". |
| 7 | Meta | Same document | postmortem | 2022-01-07 | 2026-09-26 | https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/final_update.md | Automated recovery worked once built: 8 unattended recoveries in about a week, with a 14-day-unattended goal to dissolve the on-call | (E) "we were able to automatically recover from 8 hardware failures between Christmas and New Years, greatly reducing the burden of the on-call. We have set a January infrastructure goal of training at 175B scale for 14 days without human intervention, which will allow us to officially dissolve the role of the on-call." |
| 8 | Meta | OPT-175B chronicles, "10% through" | postmortem | 2021-11-17 | 2026-09-26 | https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/10_percent_update.md | Once training settings stabilised, every remaining restart was hardware or infrastructure | (E) "since then, the only restarts we've had to make were all related to hardware issues (missing GPUs on instances, training randomly hanging after including a new node, ECC errors, partial checkpoint upload after hardware error, CUDA errors, NCCL errors, etc.)." |
| 9 | BigScience | BLOOM tr11-176B-ml chronicles, entry 2022-03-21/22 | postmortem | 2022-03-21 | 2026-09-26 | https://github.com/bigscience-workshop/bigscience/blob/master/train/tr11-176B-ml/chronicles.md | A GPU crash 2 iterations before a save cost 7.3 hours, and the checkpoint interval was retuned from fear of loss, with the save-cost arithmetic written out | (E) "we lost 7.3h of work, it failed 2 iterations before checkpoint saving time :(" and (E) "Will switch to more frequent checkpoint saving of 200 iterations and will lower it further if the hardware failures continue. With 2.3TB checkpoint size and 40 secs to save a checkpoint we don't want to do it too often." then (E) "Switched to SAVE_INTERVAL=100 ... so now with 100 iterations per checkpoint we would lose at most 3h." |
| 10 | BigScience | Same file, entry "2022-04-XX repeated hardware crashes" | postmortem | 2022-04 | 2026-09-26 | https://github.com/bigscience-workshop/bigscience/blob/master/train/tr11-176B-ml/chronicles.md | Steady-state hardware attrition was one GPU death per week, absorbed by auto-restart; hangs were the failure that defeated the automation, and a kill-switch file had to be invented around scheduler permissions | (E) "Every week or so we have hardware issues where one of the gpus dies. Most of the time the SLURM job auto-restarts and we lose at most 3 hours of training" and (E) "Sometimes the training gets stuck despite a crashing process and it won't quit" and (E) "we use a special kill switch file which when created gracefully exits the training after saving a checkpoint." |
| 11 | BigScience | Same file, entry "2022-04-28 sudden 5% drop in throughput" | postmortem | 2022-04-28 | 2026-09-26 | https://github.com/bigscience-workshop/bigscience/blob/master/train/tr11-176B-ml/chronicles.md | A fail-slow GPU cost 5% of fleet throughput and was found by excluding nodes three at a time; the node-level benchmark showed only 2.5% deviation | (E) "The performance of the system dropped from 149 to 140 TFLOPs and remained there." and (E) "Basically I had to exclude 3 nodes at a time to first find the group of nodes that was causing the slowdown, and then binary search through the nodes of that group" and (E) "Interestingly the benchmark run by Remi on that node showed only 2.5% difference (303.6s vs around 296.5 s) - and not 5%". |
| 12 | BigScience | Same file, entry "2022-04-30 hanging at eval" | postmortem | 2022-04-30 | 2026-09-26 | https://github.com/bigscience-workshop/bigscience/blob/master/train/tr11-176B-ml/chronicles.md | An 18-hour loss from a hang that no timeout caught: GPUs at 100%, NCCL silent for 3+ hours | (E) "So it has been stuck for at least 3 hours. No timeouts from NCCL or distributed, just gpus spinning at 100%." and (E) "For some reason NCCL wasn't timing out either! even after 3 hours of not being able to broadcast." and (E) "by the time I found the cause and rolled back we lost about 18h of training :(". |
| 13 | BigScience | Same file, entry "2022-05-18 slurm doesn't quit on some crashes" | postmortem | 2022-05-18 | 2026-09-26 | https://github.com/bigscience-workshop/bigscience/blob/master/train/tr11-176B-ml/chronicles.md | The crash-to-restart chain itself failed routinely; the fix was to force errors to be fatal and let the scheduler kill the job | (E) "one of the recurrent problems we have is one when the training crashes, but SLURM doesn't quit - so unless someone is around to notice it the training just hangs there in limbo and doesn't get a chance to restart." and (E) "# force crashing on nccl issues like hanging broadcast / export NCCL_ASYNC_ERROR_HANDLING=1". |
| 14 | PyTorch | Issue #76287, "torch.elastic fails to shutdown despite crashed processes" (filed by the BLOOM operator) | source | 2022-04-24 | 2026-09-26 | https://github.com/pytorch/pytorch/issues/76287 | The restart machinery (torch.elastic) failed to kill workers after a node crash, leaving the job in limbo for 8 hours; still open as of 2026-09 | (E) "It should have killed all workers so that SLURM could have exited the job which would have restarted anew recovering from the problem, but it was just hanging in an infinite state of logging for 8 hours before I noticed the problem and manually killed the slurm job" (state: open, checked 2026-09-26). |
| 15 | Hugging Face (Stas Bekman) | ML Engineering, "Fault Tolerance" chapter | blog | 2023, maintained through 2026-09 | 2026-09-26 | https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md | The BLOOM checkpoint arithmetic, generalised: 40s saves every 3h cost 0.37% of the run; 5x slower IO would make it 2% | (E) "it took only 40 seconds to save a 2.3TB checkpoint written concurrently on 384 processes. We saved a checkpoint approximately every 3 hours ... that is an additional 8 hours was spent just saving the checkpoints ... or ~0.37% of the total training time ... Now say if the IO were to be 5 times slower ... that would have become 2% of the training time". |
| 16 | Hugging Face (Stas Bekman) | Same chapter | blog | 2023, maintained through 2026-09 | 2026-09-26 | https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md | Early-life accelerator failure rates reach ~10%, so spare capacity is a design requirement, not an optimisation | (E) "There can be as large as 10% failure rate early on for new accelerators and still pretty high percentage of failures at later stages. Remember that if you have 8 accelerators, even one failing accelerator is like all 8 failing from the perspective of the training program." |
| 17 | Hugging Face (Stas Bekman) | Same chapter | blog | updated 2026 | 2026-09-26 | https://github.com/stas00/ml-engineering/blob/master/training/fault-tolerance/README.md | The checkpoint-free alternative: with replicated weights and a standby node, recovery loses one iteration instead of hundreds; torchft is the closest open implementation | (E) "This approach is superior to file system checkpointing saving because, you only ever lose one iteration, whereas with file system checkpointing you may lose hundreds of iterations." and (E) "It treats each training step as a fault boundary across replica groups, coordinates health with a lighthouse/quorum service, and can heal a failed group by live state transfer from a healthy peer". |
| 18 | PyTorch | ProcessGroupNCCL.hpp / .cpp at main |
source | 2026-09 | 2026-09-26 | https://github.com/pytorch/pytorch/blob/main/torch/csrc/distributed/c10d/ProcessGroupNCCL.hpp | The framework's stall detection is two nested deadlines: a 10-minute default collective timeout, and an 8-minute heartbeat on the watchdog itself, after which the process is killed | (E) "constexpr auto kProcessGroupNCCLDefaultTimeout = std::chrono::milliseconds(10 * 60 * 1000);" (.hpp line 164-165); (E) "heartbeatTimeoutInSec_ = getCvarInt(TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC, 60 * 8 /8 Mins/);" (.cpp lines 1772-1773); (E) comment: "Enable monitoring thread which aborts the process when the ProcessGroupNCCL Watchdog thread gets stuck and no heartbeat is detected after TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC ... It is Useful to prevent jobs being stuck for a prolonged time than necessary tying up cluster resources." |
| 19 | PyTorch | Issue #121055, "Flight Recorder dumps fail when TORCH_NCCL_ASYNC_ERROR_HANDLING=1" | source | 2024-03-01, closed not planned 2026-05-05 | 2026-09-26 | https://github.com/pytorch/pytorch/issues/121055 | The diagnostic dump can fail under exactly the error-handling setting operators are told to enable; filed by a maintainer, closed as not planned | (E) "calling commAbort from inside watchdog thread leads to exit, 'broken promise' exception from async dump thread, and no dump file." |
| 20 | Meta / PyTorch | torchft repository README | source | current at 2026-09; first PyPI release 2024-10-13 | 2026-09-26 | https://github.com/meta-pytorch/torchft | Per-step fault tolerance: quorum on every step, so membership changes without stopping the world; recovery is live transfer from a healthy peer | (E) "This allows for membership changes at the training step granularity which can greatly improve efficiency by avoiding stopping the world training on errors." and (E) "Checkpoint transports that can be used to do live recovery from a healthy peer when doing scale up operations." |
| 21 | Meta / PyTorch | "Fault Tolerance for Large Scale Training", PyTorch Conference 2024 poster (PDF in the torchft repo) | talk | 2024-09 | 2026-09-26 | https://github.com/meta-pytorch/torchft/blob/main/media/fault_tolerance_poster.pdf | The detection layer Meta ships: flight recorder classifies deadlocks, stragglers, mismatched collectives and timeouts; a per-node HTTP WorkerServer exposes dumps, py-spy and fault injection to an external monitor | (E) "Flight Recorder is a cutting-edge PyTorch feature that captures and logs records NCCL operations, providing valuable insights for identifying bad hosts and detecting errors including: Deadlocks ... Stragglers: slow/underperforming ranks ... Mismatched collectives ... NCCL timeouts and comm failures" and (E) "On failure we use a customized comm library to gracefully handle errors and continue training on the next batch without downtime." |
| 22 | Meta / PyTorch | torchtitan experiments/torchft README |
source | current at 2026-09 | 2026-09-26 | https://github.com/pytorch/torchtitan/blob/main/torchtitan/experiments/torchft/README.md | The recovery unit is the replica group, and one detail shows the care required: the surviving group with the smallest id owns checkpointing | (E) "With TorchFT, we can tolerate one replica group failure, ensuring that the training process continues uninterrupted." and (E) "Note that the alive replica group with the smallest replica ID will perform checkpointing saving." |
| 23 | NVIDIA | nvidia-resiliency-ext README | vendor | current at 2026-09; first PyPI release 2024-10-15 | 2026-09-26 | https://github.com/NVIDIA/nvidia-resiliency-ext | The vendor's packaged answer has the same four organs: hang detection, in-job restart, async + local checkpointing, straggler detection | (E) "Detection of hung ranks." and (E) "Restarting training in-job, without the need to reallocate SLURM nodes." and (E) "NVRx maximizes goodput by enabling system-wide health checks, quickly detecting faults at runtime and resuming training automatically." |
| 24 | Microsoft / Argonne | DeepSpeed PR #5763, "Add DataStates-LLM: Asynchronous Checkpointing Engine Support" | source | opened 2024-07-10, closed unmerged 2024-11-15 | 2026-09-26 | https://github.com/deepspeedai/DeepSpeed/pull/5763 | An async checkpointing engine offered to DeepSpeed sat four months and was closed unmerged on staleness; checkpointing improvements fragment per framework rather than accumulate | (E) maintainer: "I'm going to close this PR now as stale, but we would appreciate the contribution if you are able to come back to this and resolve the merge conflicts?" |
| 25 | Ant Group | DLRover README | casestudy | current at 2026-09 (claim first published 2023-08) | 2026-09-26 | https://github.com/intelligent-machine-learning/dlrover | A measured production goodput delta from fault-tolerance automation alone: 69% to 95% on a thousands-of-GPUs job | (E) "With fault tolerance, the goodput of GLM-65B training on thousands of GPUs increased from 69% to 95%. The goodput is the time spent computing useful new steps over the elapsed time of the training job." |
| 26 | Ant Group | DLRover "Flash Checkpoint" design blog (in-repo) | blog | 2024-01 | 2026-09-26 | https://github.com/intelligent-machine-learning/dlrover/blob/master/docs/blogs/flash_checkpoint.md | Tiered checkpointing collapses the save cost: synchronous copy to host shared memory, async persistence, reload from memory on process restart | (E) "the wasted time to checkpoints was reduced by about 5 times, where the persistence time was reduced by about 70 times, effectively increasing the goodput of training from 90% to 95%." and (E) "If the reason of failure is not fault hardware, DLRover can simply restart the training process, at which point the checkpoint can be directly loaded from the host memory". |
| 27 | Ant Group | DLRover "Unified Failover" design document | adr | current at 2026-09 | 2026-09-26 | https://github.com/intelligent-machine-learning/dlrover/blob/master/docs/design/unified-failover.md | The failover mechanism is a first-class design document, not an ops afterthought; per-scenario failover paths are enumerated | (E) "This document describes the failover (fault tolerance) mechanisms in DLRover Unified" (title and overview). |
| 28 | Orbax emergency checkpointing, checkpoint_manager.py at main |
source | current at 2026-09 | 2026-09-26 | https://github.com/google/orbax/blob/main/checkpoint/orbax/checkpoint/experimental/emergency/checkpoint_manager.py | Google's JAX path institutionalises two-tier checkpointing: node-local saves every 10 steps by default, a persistent tier less often, restore preferring the local copy | (E) "save_interval_steps: The interval at which checkpoints should be saved to local storage. Ensures checkpoints will only be saved every m steps. Defaults to 10." and module docstring: (E) "LocalCheckpointManager is designed to be created and called across every process within non-primary slices." | |
| 29 | Meta | Llama 3.1 model card (llama-models repo) | vendor | 2024-07 | 2026-09-26 | https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md | The scale that fixes the stakes: 39.3M H100-hours cumulative for the Llama 3.1 family, so every hour of downtime is thousands of GPU-hours | (E) "Training utilized a cumulative of 39.3M GPU hours of computation on H100-80GB (TDP of 700W) type hardware". |
| 30 | Meta / NVIDIA / Ant Group | PyPI release records for torchft, nvidia-resiliency-ext, dlrover | vendor | 2023-01 to 2026-09 | 2026-09-26 | https://pypi.org/project/torchft/ | The tooling wave is dated by the registry: torchft 0.1.0 on 2024-10-13 and nvidia-resiliency-ext 0.1.3 on 2024-10-15, two days apart; dlrover first published 2023-01-16 | (R) upload timestamps from the PyPI JSON API, checked 2026-09-26: torchft first release 2024-10-13, latest 0.2.0 2026-08-27; nvidia-resiliency-ext first 2024-10-15, latest 0.7.0 2026-09-16; dlrover first 2023-01-16. See also https://pypi.org/project/nvidia-resiliency-ext/ and https://pypi.org/project/dlrover/ |
Derived figures used in the guide
- OPT restart rate: ~90 restarts (row 5) over the experiment-12 lineage, launched 2021-11-11 (row 8's document) and completed 2022-01-06 (row 5), is 56 days: ~1.6 restarts per day average across the run. Marked (D) wherever used.
- BLOOM worst-case loss per failure after retuning: interval 100 iterations at about 105 s per iteration (both in row 9/11 quotes) is about 3 hours of exposure against a 40-second save cost, a roughly 270:1 ratio between what a failure costs and what avoiding it costs per save. Marked (D).
- OPT single-failure cost in GPU-hours: a mid-interval failure on 1024 GPUs with recovery measured in hours (rows 3, 5) is thousands of GPU-hours per event; at the 7.3-hour BLOOM loss (row 9) on 384 GPUs, ~2,800 GPU-hours for one ECC error. Marked (D).