GPU Health Checks and Silent Data Corruption

During a 54-day snapshot of Llama 3 405B pre-training on 16,000 H100s, the job was interrupted 466 times: 47 planned, 419 unexpected. About 78 percent of the unexpected interruptions were confirmed or suspected hardware issues, GPU issues alone accounted for 58.7 percent, and six were attributed to silent data corruption (Grattafiori et al., 2024, The Llama 3 Herd of Models, arXiv:2407.21783). Effective training time still exceeded 90 percent, because detection and replacement were largely automated.

Fault tolerance for long training runs covers checkpoint intervals and recovery. This concept is about the other half, deciding which machines are fit to run a job at all, and catching the ones that fail without crashing.

Failure scales with node count

Meta's study of two research clusters, 11 months, 4 million jobs and over 150 million A100 GPU-hours, found job mean time to failure falls in inverse proportion to job size, fitting

where \(r_f\) is the failure rate per node-day. With their RSC-1 estimate \(r_f = 6.5 \times 10^{-3}\), a 16,384-GPU job on 2,048 eight-GPU nodes expects \(2{,}048 \times 0.0065 \approx 13.3\) failures a day, an MTTF of about 1.8 hours, matching their projection (Kokolis et al., 2024, Revisiting Reliability in Large-Scale Machine Learning Research Clusters, arXiv:2410.21680). The same paper states the operational consequence plainly: under gang scheduling, one bad node spoils the whole job. That is the argument for screening nodes before placement rather than relying only on recovery afterwards.

Layers of checking

Burn-in stresses new or repaired nodes before they join the pool: extended diagnostics, memory tests, sustained full-power load to shake out thermal and power faults, and multi-node collective tests over every NIC. Longer burn-in catches more infant-mortality failures and leaves expensive hardware idle; the right duration is an empirical trade that operators tune from their own repair data.

Periodic and between-job checks repeat cheaper tests: GPU and NIC link state, ECC and XID error counters, filesystem mounts, driver and firmware versions. A failed check drains the node rather than letting it absorb the next job.

Lemon detection handles nodes that pass every check and keep failing jobs. Kokolis et al. built signals from per-node history, such as XID counts, repair tickets and jobs that excluded the node, and identified 40 lemon nodes with over 85 percent accuracy, cutting the failure rate of 512-GPU-plus jobs from 14 percent to 4 percent. The lesson is that some faults are only visible statistically, across many jobs.

In-job telemetry watches what checks cannot see: per-rank step time, collective timeouts, loss spikes and NaNs.

Stragglers

Synchronous training runs at the speed of its slowest rank: step time is roughly \(\max_i t_i\) plus communication, so one GPU throttling 20 percent slows every GPU 20 percent. Llama 3's authors describe hardware causing "still-functioning but slow stragglers that are hard to detect", and note even the diurnal effect of temperature on voltage and frequency scaling moved throughput 1-2 percent. A robust detector flags rank \(i\) when

for \(w\) consecutive steps, where MAD is the median absolute deviation and the 1.4826 factor scales it to a standard deviation under normality. Medians resist the straggler contaminating its own baseline, and the persistence window \(w\) suppresses one-off stalls from checkpoint writes.

Silent data corruption

A silent data corruption is a wrong result with no error raised. Meta found hundreds of CPUs producing SDCs across hundreds of thousands of machines, undetectable by hardware error reporting (Dixit et al., 2021, Silent Data Corruptions at Scale, arXiv:2102.11245). Google reported "a few mercurial cores per several thousand machines" (Hochschild et al., 2021, Cores that don't count, HotOS). For Gemini, Google expected SDC events to affect training "every week or two", and responded with deterministic replay to isolate incorrect computations, proactive SDC scanners on idle machines and hot standbys (Gemini Team, 2023, Gemini: A Family of Highly Capable Multimodal Models, arXiv:2312.11805).

Detection options in practice: known-answer tests that compute a fixed workload and compare against a stored result; replaying a suspicious step on different hardware; and, where replicas should hold identical parameters, as in plain data parallelism, periodically comparing parameter checksums across replicas. A mismatch localises the fault to a replica before the corruption propagates through the next all-reduce.

When it breaks

Aggressive exclusion starves capacity. Every false positive drains a working node. Kokolis et al. note manual, aggressive exclusion does not scale and can starve capacity, which is why statistical lemon detection beats engineers' blocklists.

SDC detection is sampling. Known-answer tests exercise a fraction of instructions and operating conditions; a core that fails only at one temperature or on one kernel path passes. Screening lowers the rate and does not certify a node.

Checksums break with sharding. Under fully sharded or tensor-parallel training no two ranks hold the same bytes, so cross-replica comparison needs deliberate redundancy or replay instead.

Corruption can precede detection by hours. A node flagged now may have produced bad gradients since its last clean check, so the rollback target is the last checkpoint before the first suspect step, not the most recent one.