At Sixteen Thousand GPUs, Something Is Always Broken: Failures, Stragglers and Silent Data Corruption in Training Clusters

In a 54-day window of Llama 3 405B pre-training, Meta's job stopped 466 times. Forty-seven stops were planned. The other 419 were not: 148 attributed to faulty GPUs, 72 to HBM3 memory, 54 to software bugs, 35 to network switches and cables, and six to silent data corruption, GPUs that returned wrong numbers without raising an error (Dubey et al., 2024, The Llama 3 Herd of Models, arXiv:2407.21783). That is an unexpected interruption about every 3.1 hours on up to 16,384 H100s. The paper still reports more than 90% effective training time, with significant manual intervention needed only three times.

The question at this scale is how cheaply failure can be absorbed. The arithmetic is fifty years old, and it contradicts the folklore: past a certain scale the checkpoint interval hardly matters, while restart cost grows linearly with the cluster.

Why this matters: Every lab training at frontier scale pays a reliability tax measured in percentage points of a cluster that costs hundreds of thousands of dollars a day. Knowing where it comes from (checkpoint writes, lost work, restarts, stragglers, and corrupted math no error counter sees) separates a cluster that delivers 90% goodput from one that delivers 70% on identical hardware.

TL;DR

• Job MTTF falls in inverse proportion to job size: Meta measured 47.7 days at 8 GPUs and 7.9 hours at 1,024, and projects 1.8 hours at 16,384 and 0.23 hours at 131,072 (Kokolis et al., 2025).
• Llama 3's 419 unexpected interruptions imply about 3.8 failures per thousand node-days, between Meta's two research clusters (2.34 and 6.50).
• The Young/Daly interval \(\sqrt{2\delta M}\) sits in a flat valley: at a 3.1-hour MTBF and 5-minute checkpoints, intervals of 30, 43 and 60 minutes give 71.9%, 72.4% and 71.1% goodput.
• Checkpoint overhead grows with \(\sqrt{N}\), restart overhead with \(N\). At 131,072 GPUs a 15-minute restart alone eats about 65% of wall-clock time.
• Cutting checkpoint stalls from 5 minutes to 30 seconds and restarts from 15 minutes to 3 lifts modelled goodput at 16K GPUs from 72% to 91%.
• Stragglers leak more than most teams assume: in 3,079 ByteDance jobs, 42.5% straggled and 10.4% of allocated GPU-hours were wasted, mostly from software imbalance rather than bad hardware (Lin et al., 2025).
• Silent data corruption is rare per chip and routine per cluster: Google expected it to hit Gemini training "every week or two" (Gemini Team, 2023).

At a Glance

flowchart LR
    subgraph Sources["What goes wrong"]
        F1["Fail-stop: GPU, HBM, NIC"]
        F2["Stragglers: slow but alive"]
        F3["Silent data corruption"]
    end
    subgraph Detect["How it is noticed"]
        D1["Watchdogs and heartbeats"]
        D2["Per-rank step timing"]
        D3["Replay, checksums, scanners"]
    end
    subgraph Recover["What it costs"]
        R1["Rework since checkpoint"]
        R2["Restart and reload"]
        R3["Rollback past corruption"]
    end
    G["Goodput: useful time over wall clock"]
    F1 --> D1 --> R2
    F2 --> D2 --> G
    F3 --> D3 --> R3
    D1 --> R1
    R1 --> G
    R2 --> G
    R3 --> G

    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    class F1,F2,F3 rose
    class D1,D2,D3 purple
    class R1,R2,R3 amber
    class G teal

Fail-stop faults are loud and frequent. Stragglers never stop the job, so no interruption table records them. Silent corruption is rarest and costliest, because its price is a rollback to a point you may not know.

From Supercomputer Dumps to GPU Fleets

In 1974 John Young, writing about mainframe batch jobs, showed that the checkpoint interval minimising expected waste is roughly the square root of twice the product of checkpoint time and mean time between failures (Young, 1974, CACM 17(9)). John Daly at Los Alamos refined it in 2006, adding restart time and higher-order terms, with a three-term approximation that never costs more than 0.2% of solution time versus the exact optimum (Daly, 2006, Future Generation Computer Systems 22(3)). Deep learning mostly ignored this while jobs fit on a few nodes with MTTFs of weeks.

Then training became one synchronous job across thousands of accelerators, and processors turned out not always to fail by stopping. Meta reported in 2021 that testing across hundreds of thousands of machines had found hundreds of CPUs producing reproducible silent data corruptions with nothing in the hardware error logs (Dixit et al., 2021, arXiv:2102.11245). Google described "mercurial" cores at "on the order of a few mercurial cores per several thousand machines" (Hochschild et al., 2021, Cores that don't count, HotOS).

Then the logbooks appeared. OPT-175B, on 992 A100s over about two months, recorded at least 35 manual restarts, over 100 hosts cycled and an estimated 70-plus automatic restarts (Zhang et al., 2022, arXiv:2205.01068), and its team repeatedly rolled back past loss divergences. By 2024 the numbers came as tables, and the question had become automation.

timeline
    title Failure at scale, from mainframes to frontier clusters
    1974 : Young derives the square-root checkpoint interval
    2006 : Daly adds restart cost and higher-order terms
    2021 : Meta and Google report silent data corruption in CPU fleets
    2022 : OPT-175B logbook records 35 manual restarts on 992 A100s
         : Meta compares out-of-production and in-production SDC testing
    2023 : Gemini expects SDC every week or two
         : In-memory replicas lift goodput from 85 to 97 percent
    2024 : MegaScale runs 12288 GPUs through over 100 restarts
         : Llama 3 publishes 466 interruptions in 54 days
         : Meta fits research-cluster MTTF against job size
    2025 : ByteDance finds stragglers waste 10.4 percent of GPU hours
         : Real-world SDC shown to cause loss spikes in LLM training

[IMAGE: Horizontal bar chart of Llama 3 405B's 18 root-cause categories sorted by count, coloured by category (GPU rose, host slate, network amber, software purple). Annotate the Faulty GPU bar "148 events; stated 30.1%, computes to 35.3%". Caption: "Where 419 unexpected interruptions came from in 54 days."]

How Failure Turns Into Lost Goodput

The failure rate is a per-node constant times the job size

Kokolis and colleagues analysed 11 months of two Meta research clusters, 4 million jobs and over 150 million A100 GPU-hours, and fitted job MTTF with one parameter, the failure rate per node-day \(r_f\) (Kokolis et al., 2025, Revisiting Reliability in Large-Scale Machine Learning Research Clusters, HPCA, arXiv:2410.21680):

They measured \(r_f = 6.50\) per thousand node-days on RSC-1 and \(2.34\) on RSC-2, where GPUs were swapped a third as often. The job is a series system: any node failing fails it, and independent rates add.

Llama 3's table lets us back out the same parameter for production pre-training. With 419 unexpected interruptions over 54 days on 2,048 eight-GPU nodes,

between the two research clusters. This is an estimate (Llama 3 ran on "up to" 16K GPUs, and software bugs are not node failures), but it gives \(M \approx 54 \times 24 / 419 \approx 3.1\) hours.

The cost of one failure

A failed synchronous job loses three kinds of time: work since the last checkpoint, the restart (detect, drain the bad node, allocate a replacement, launch, initialise communicators, load state), and, continuously, the stalls spent writing checkpoints.

Let \(\tau\) be useful compute between checkpoints, \(\delta\) the time a checkpoint blocks training, \(R\) the restart time and \(M\) the mean time between failures. A failure lands uniformly within a segment of length \(\tau + \delta\), destroying half of one on average. To first order, wall-clock time per unit of useful work is

The first term falls as checkpoints get rarer; the second rises. Setting the derivative in \(\tau\) to zero, \(-\delta/\tau^2 + 1/(2M) = 0\), gives Young's result:

At the optimum the checkpoint and rework terms are each \(\sqrt{\delta/2M}\), so the minimum overhead is

This expression holds the whole argument. Substitute \(M = 1/(N_{\text{nodes}} r_f)\):

Checkpoint waste grows with the square root of cluster size; restart waste grows linearly. Doubling the cluster multiplies the first term by 1.41 and the second by 2, so at some scale restart time overtakes checkpointing as the dominant cost however well the interval is tuned. Kokolis et al. reach the same place with their expected-ETTR model: for a 12,288-GPU job to reach an ETTR near 0.9, RSC-1's failure rate would need to fall from 6.50 to about 1, or checkpoint write overhead to the order of 10 seconds.

Daly's exact model

The first-order form overstates goodput once \(\tau\) and \(R\) are not small against \(M\), because failures also strike during restarts and rework. Treating failures as a Poisson process, Daly derives the expected wall time for \(T_s\) of useful work exactly:

The exponentials charge for interrupted restarts and repeated attempts at a segment. Goodput is \(G = T_s / T_w\). Its minimiser has the perturbation series

with \(\tau_{\text{opt}} = M\) when \(\delta \geq 2M\). Subtracting \(\delta\) reflects that the checkpoint itself is exposed to failure. For every scenario in this post, Young's and Daly's intervals differ in goodput by under half a point. The optimum is shallow, which is the first practical lesson: tune the interval once, roughly, and spend the effort on \(\delta\) and \(R\).

Stragglers are a separate tax

A straggler costs nothing visible; the job simply runs at its slowest rank's speed, because every rank waits at the collective:

Two mechanisms feed the max. One is a persistently slow device. MegaScale found specific hosts taking about 10% longer on identical forward computations, and a heat map showing roughly 0.5% of machines substantially slower, with efficiency "predominantly determined by the slowest machine's performance" (Jiang et al., 2024, MegaScale, NSDI, arXiv:2402.15627). If 0.5% of 2,048 nodes are slow and scattered independently, the chance a job avoids all of them is \(0.995^{2048} \approx e^{-10.3}\), effectively zero.

The other needs no fault. If each rank's compute time has independent jitter of standard deviation \(\sigma\), the expected maximum of \(N\) roughly Gaussian draws grows as \(\sigma\sqrt{2\ln N}\): about \(3.9\sigma\) at 2,048 ranks, \(4.4\sigma\) at 16,384. Scale alone turns per-GPU noise (garbage collection, uneven sequence lengths, pipeline imbalance) into job-level slowdown.

Silent corruption breaks the model

Everything above assumes failures announce themselves. An SDC does not. A GPU with a marginal execution unit produces a wrong partial sum, the all-reduce faithfully averages it into every replica, and the corruption now lives in the whole job's weights. Its cost is not \(R + \tau/2\) but \(R\) plus the detection latency \(L\), and rollback is only possible if a checkpoint older than \(L\) survives: keeping \(K\) checkpoints at interval \(\tau\) protects only against corruption detected within \(K\tau\). Llama 3 lists six SDC interruptions in 54 days. Those are the ones that were caught.

[IMAGE: Goodput versus checkpoint interval (0 to 240 minutes) at MTBF 186 minutes, three curves for checkpoint cost 5 minutes, 30 seconds and 10 seconds, restart 15 minutes, each with its Young optimum marked. Caption: "A wide, flat valley. Lifting the curve (cheaper checkpoints) matters far more than finding its lowest point."]

Seeing It in Motion

A failure noticed after 20 minutes has added 20 minutes to \(R\), so detection is where engineering effort goes. The sequence below composites the mechanisms MegaScale and Llama 3 describe: per-node heartbeats, a collective watchdog, automated diagnostics, eviction and resume.

sequenceDiagram
    participant G as GPU rank 4711
    participant A as Node agent
    participant C as Job controller
    participant S as Scheduler
    participant K as Checkpoint store
    G->>G: NVLink stall inside collective kernel
    A->>C: Heartbeat shows RDMA traffic stopped
    Note over G,C: NCCL watchdog fires and dumps flight recorder
    C->>A: Run intra-host and NCCL self-tests
    A-->>C: Node fails loopback bandwidth test
    C->>S: Evict node, request replacement
    S-->>C: Spare node allocated
    C->>K: Load latest sharded checkpoint
    K-->>C: State restored on all ranks
    Note over C,K: Rework since checkpoint, then steady state

MegaScale reports diagnostics under 10 minutes and catch-up within 15 minutes of the latest checkpoint. The high-variance steps are noticing, and finding which of 2,048 nodes is to blame; Llama 3's team relied on PyTorch's NCCL flight recorder because NVLink failures often surface as stalled kernels "without returning a clear error code."

Clusters that sustain high goodput treat nodes as moving through states, and place work only on healthy ones:

stateDiagram-v2
    [*] --> BurnIn
    BurnIn --> Healthy: passes stress and NCCL tests
    BurnIn --> Repair: fails diagnostics
    Healthy --> Suspect: health check or step-time alarm
    Suspect --> Healthy: rechecks pass
    Suspect --> Drained: confirmed fault
    Drained --> Repair
    Repair --> BurnIn: parts replaced
    Healthy --> Lemon: repeated job failures
    Lemon --> Repair
    Repair --> Retired: repeat offender
    Retired --> [*]

Burn-in and self-tests feed the Healthy state. MegaScale's version is concrete: a loopback test from every RDMA NIC to memory nodes and GPUs across all link combinations, an RNIC-to-RNIC test, an all-to-all NCCL test inside the node, then all-reduce with neighbours. Smaller operators use the open-source nccl-tests for the same job. The Lemon state is the non-obvious one. Kokolis et al. found nodes that pass every check yet fail jobs above chance; from signals such as how many distinct jobs had excluded a node, its unique XID errors and its repair tickets, they flagged lemons making up 1.2% of RSC-1, and removing them cut the failure rate of 512-GPU-plus jobs from 14% to 4%. Some faults live only in a machine's history.

[IMAGE: Screening funnel for a hypothetical 2,100-node pool: burn-in, intra-host tests, inter-node NCCL tests, 2,048 placed plus 52 spares, with side arrows showing where infant mortality, bad NICs, bad cables, lemons and SDC are typically caught. Caption: "Each layer catches a different class of fault; what escapes all of them becomes an interruption."]

By the Numbers

Sources: values are read from the cited papers' text and tables. The Llama 3 MTBF and \(r_f\) are derived by dividing its counts by its window and node count, and are estimates. One discrepancy: Table 5 gives 148 faulty-GPU events as 30.1%, but 148/419 is 35.3%; every other row equals its count over 419, the percentages sum to 94.9%, and the text's 58.7% GPU share sums the stated percentages. Gemini's SDC cadence is Google's expectation, not a count.

[IMAGE: Log-log plot of job MTTF against job size, 8 to 131,072 GPUs, with slope minus one lines for r_f = 2.34, 3.8 and 6.50 per thousand node-days, plus measured points at 8 and 1,024 GPUs and Llama 3's 3.1 hours at 16,384. Caption: "A series system: every doubling of the job halves its time between failures."]

A Concrete Example

A 16,384-GPU pre-training job on 2,048 eight-GPU nodes, at Llama 3's implied failure rate. What does each improvement buy?

Step 1: failure rate. \(419 / 54 = 7.76\) interruptions a day, so \(M = 1{,}440 / 7.76 \approx 186\) minutes.

Step 2: naive baseline. A synchronous checkpoint blocks for \(\delta = 5\) minutes and a restart takes \(R = 15\) (Kokolis et al. cite restart overheads of 5 to 20 minutes; MegaScale's catch-up is within 15). Young gives \(\tau^{*} = \sqrt{2 \times 5 \times 186} = \sqrt{1860} \approx 43.1\) minutes; Daly's formula gives 39.9.

Step 3: decompose the waste. Checkpoint writes \(5 / 43.1 = 11.6\%\); rework \((43.1 + 5)/(2 \times 186) = 12.9\%\); restarts \(15/186 = 8.1\%\). First-order overhead 32.6%, goodput \(1/1.326 = 75.4\%\). Daly's exact model, charging for failures during restarts and rework, gives 72.4%.

Step 4: the interval barely matters. In the exact model, 30 minutes gives 71.9% and 60 minutes 71.1%; only 120 (62.1%) and 240 (43.6%) hurt.

Step 5: cheap checkpoints. MegaScale's two-stage design copies state to pinned host memory in seconds and uploads to HDFS in the background. With a 30-second stall, \(\tau^{*} = \sqrt{2 \times 0.5 \times 186} = 13.6\) minutes, checkpoint waste 3.7%, rework 3.8%, restarts still 8.1%. Goodput 85.7%.

Step 6: cheap restarts. Hot spares, cached images and fast communicator setup (MegaScale cut initialisation from 1,047 seconds to under 5 on 2,048 GPUs) bring \(R\) to 3 minutes; restart waste falls to 1.6%. Goodput 91.4%, the regime Llama 3 and MegaScale report, reached only by attacking \(\delta\) and \(R\).

Step 7: scale eight-fold to 131,072 GPUs. \(M = 186/8 = 23.2\) minutes. With steps 5 and 6, goodput is 70.9%; restart waste alone rises from 1.6% to 12.9%. With the step 2 baseline it collapses to 24.8%, and restarts alone take \(15/23.2 = 65\%\) of wall clock. The linear term has won.

Step 8: price it. At an assumed, illustrative $2 per GPU-hour the 16K cluster costs about $786,000 a day, so the 19 points between steps 3 and 6 are worth roughly $150,000 a day, or $8 million over 54 days.

These are model outputs assuming exponential failures; real failures cluster.

Where It Breaks

The exponential assumption

Young and Daly assume memoryless failures. Infant mortality makes new and repaired nodes fail more, the argument for burn-in; lemon nodes fail repeatedly, correlating failures. Planned maintenance is not random at all: Llama 3's automated upgrades caused "at least one training interruption daily", but a planned stop can checkpoint first and lose no work.

Detection latency hides inside \(R\)

The model treats \(R\) as constant, but its largest component is often the time before anyone knows the job is dead. A rank stalled in an NVLink load does not crash; every other rank waits with it at the next collective, and default timeouts are long. Kokolis et al. note that once health checks remove obvious failures, the rest "often manifest with proximal failures that do not immediately suggest a root cause", NCCL timeouts among the commonest. A timeout is a symptom on every rank, not a diagnosis on one; MegaScale's RDMA-traffic heartbeats and Llama 3's flight recorder exist to localise it.

Stragglers hide in averages

In ByteDance's trace, 42.5% of jobs ran at least 10% slower than a no-straggler simulation, and about 1% wasted at least 45% of their GPU-hours. Per-step slowdowns sat close to each job's overall slowdown (p90 of 1.06 normalised), so causes were persistent, not transient. Heat is one even without faults: Llama 3 saw a 1-2% diurnal throughput swing as mid-day temperatures lowered GPU clocks.

Silent corruption and the poisoned checkpoint

Real SDC usually perturbs training rather than crashing it. Ma et al. ran deterministic training on nodes a cloud provider's fleet management had swept out for SDCs. Perturbations to submodule outputs and gradients were usually small, yet models converged to different optima, and on some nodes loss spikes appeared that "can fully corrupt the model weights in some cases" (Ma et al., 2025, Understanding Silent Data Corruption in LLM Training, arXiv:2502.12340). The chain runs from one wrong device, through the all-reduce, into the checkpoint, and back in on restart. Retention sized to detection latency is the only defence at the last step.

Every detector has a cost. Known-answer tests find only faults they exercise; deterministic replay needs a deterministic stack; cross-replica checksums need bit-identical replicas, which sharded optimisers partly break. Meta's CPU experience shows the coverage trade: in-production testing reached 70% of detectable defects in about 15 days where out-of-production testing took 5 to 6 months, yet each found defects the other never did (7% and 23% unique coverage).

Fast recovery has side effects

Continuing on fewer nodes changes the global batch size and so the optimisation, a reproducibility problem if unrecorded. And the cluster is an electrical load moving in lockstep: Llama 3 reports power swings "on the order of tens of megawatts" when all GPUs pause together, which frequent checkpoints make more frequent.

[IMAGE: Two panels. Left: two loss curves with an SDC-induced spike at step 40,000, one recovering, one diverging. Right: checkpoints every 13 minutes with a shaded 90-minute "undetected corruption" window, showing only checkpoints older than the window are safe. Caption: "An SDC costs its detection latency; retention shorter than that latency cannot save you."]

Alternative Designs

The designs split by which term they attack. Just-in-time checkpointing (Gupta et al., 2024, EuroSys) exploits the fact that healthy data-parallel replicas already hold what a failed one lost, enabling recovery "with just a single minibatch iteration of work replayed by all GPUs." Here the literature disagrees. Gemini's report says high goodput at its scale "would have been impossible using the conventional approach of periodic checkpointing of weights to persistent cluster storage"; Llama 3 reports above 90% with periodic checkpoints made faster. Both can be right, because the break-even depends on \(M\), and Gemini's \(M\) is unpublished.

How It Is Used in Practice

Make failures loud. Aggressive collective timeouts, heartbeats carrying RDMA counters, and flight recorders that dump recent collectives turn a silent hang into an event with a location. Llama 3's team credits such tooling with needing significant manual intervention only three times in 54 days.

Keep history. Lemon detection is a counting problem. Acme at Shanghai AI Laboratory shows why accounting matters: infrastructure failures were 11% of failed jobs but over 82% of failed jobs' GPU time, because they strike the long, wide pre-training runs (Hu et al., 2024, NSDI, arXiv:2403.07648).

Scan for SDC continuously. Gemini combines idle-machine scanners, deterministic replay and hot standbys; smaller operators use known-answer burn tests and loss-spike alarms.

Treat stragglers as performance bugs. ByteDance found "hardware or software problems in a server were not a significant cause"; pipeline-stage imbalance, sequence-length imbalance and Python garbage collection were, and the fixes were code.

Measure goodput, not allocation. ETTR, effective training time and goodput all mean useful time over wall clock; allocation dashboards hide every cost here.

[IMAGE: Stacked bar for one 168-hour week of a 16K-GPU job: useful training, checkpoint stalls, rework, restarts, straggler slowdown and planned maintenance, using step 6 figures plus a 5% straggler segment. Caption: "Where the week goes. The largest loss is often the one no interruption table records."]

Insights Worth Remembering

1. A training cluster is a series system. Job MTTF is the per-node rate divided by node count, so a bigger cluster buys a proportionally more fragile job unless the per-node rate falls.

2. The checkpoint interval is the least important knob. Goodput is flat across a wide band around \(\sqrt{2\delta M}\); debating 30 versus 45 minutes optimises the term that barely moves.

3. Restart time is the term that grows linearly. Checkpoint waste scales as \(\sqrt{N}\), restart waste as \(N\). At frontier scale, time from failure to resumed training is the dominant reliability cost.

4. Cheap checkpoints pay twice. Lower \(\delta\) means checkpointing more often while spending less total time on it, which also shrinks rework.

5. Silent corruption turns a restart problem into a rollback problem. Its cost is detection latency, so retention is a fault-tolerance parameter and loss monitoring is a correctness mechanism.

6. Stragglers are mostly software. The largest published trace blames workload imbalance and garbage collection; health checks will not find them, per-rank timing will.

7. Interruption tables are lower bounds. They count only what was detected.

Open Questions

How many silent corruptions go undetected? Measured: Llama 3 caught six in 54 days; Google expects one every week or two. Unknown: the miss rate. Ma et al. show SDC can shift a model's optimum without a visible spike, so some likely escape, but nothing public bounds how many.

Do GPUs share the CPU SDC profile? Meta's and Google's prevalence figures are for CPUs; GPU evidence is operational, not a fleet rate. That hotter, denser accelerators corrupt more per unit of compute is plausible and unestablished.

What happens when the unit of repair is a rack? Kokolis et al. note that systems such as GB200 move repair from server to rack, so each fault removes more capacity. Public operating data is thin.

Can goodput stay above 90% past 100,000 GPUs? The model says only with second-scale \(\delta\), minute-scale \(R\) or a much lower \(r_f\). Gemini's 97% suggests in-memory recovery can; whether that generalises beyond Google's deterministic stack is unmeasured.

Is straggler waste software-dominated everywhere? ByteDance's finding cuts against MegaScale's and Llama 3's emphasis on hardware stragglers. Perhaps hardware stragglers are dramatic and rare while imbalance is mundane and constant, but only one cluster has been traced this deeply.

Sources and Further Reading

1. Dubey, A., Grattafiori, A., et al. (2024). "The Llama 3 Herd of Models." arXiv:2407.21783
2. Kokolis, A., Kuchnik, M., et al. (2025). "Revisiting Reliability in Large-Scale Machine Learning Research Clusters." HPCA 2025. arXiv:2410.21680
3. Jiang, Z., Lin, H., Zhong, Y., et al. (2024). "MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs." NSDI 2024. arXiv:2402.15627
4. Zhang, S., Roller, S., Goyal, N., et al. (2022). "OPT: Open Pre-trained Transformer Language Models." arXiv:2205.01068
5. Gemini Team, Google. (2023). "Gemini: A Family of Highly Capable Multimodal Models." arXiv:2312.11805
6. Young, J. W. (1974). "A first order approximation to the optimum checkpoint interval." CACM 17(9), 530-531. doi:10.1145/361147.361115
7. Daly, J. T. (2006). "A higher order estimate of the optimum checkpoint interval for restart dumps." Future Generation Computer Systems 22(3), 303-312. doi:10.1016/j.future.2004.11.016
8. Dixit, H. D., et al. (2021). "Silent Data Corruptions at Scale." arXiv:2102.11245
9. Dixit, H. D., Boyle, L., et al. (2022). "Detecting silent data corruptions in the wild." arXiv:2203.08989
10. Hochschild, P. H., Turner, P., Mogul, J. C., et al. (2021). "Cores that don't count." HotOS '21. PDF
11. Ma, J., Pei, H., Lausen, L., & Karypis, G. (2025). "Understanding Silent Data Corruption in LLM Training." arXiv:2502.12340
12. Lin, J., Jiang, Z., Song, Z., et al. (2025). "Understanding Stragglers in Large Model Training Using What-if Analysis." OSDI 2025. arXiv:2505.05713
13. Hu, Q., Ye, Z., Wang, Z., et al. (2024). "Characterization of Large Language Model Development in the Datacenter." NSDI 2024. arXiv:2403.07648
14. Gupta, T., Krishnan, S., Kumar, R., et al. (2024). "Just-In-Time Checkpointing: Low Cost Error Recovery from Deep Learning Training Failures." EuroSys 2024. doi:10.1145/3627703.3650085

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.