A training run on thousands of GPUs loses several nodes mid-run. How should checkpointing frequency, elastic training, straggler detection and scheduling minimise wasted compute, and what does each checkpoint cost?

Why failure is the normal case

At thousands of accelerators running for weeks, hardware failure is expected rather than exceptional. Published accounts of large training runs — including Meta's descriptions of LLaMA-scale training — report failures at a rate of multiple interruptions per day, dominated by GPU and network faults.

Synchronous data-parallel training means one failed node halts the entire run, because every step requires a collective operation across all ranks. The blast radius of one GPU is the whole cluster, which is what makes this problem structurally different from ordinary distributed computing.

Checkpointing, and its real cost

A checkpoint writes model parameters, optimiser state and data-loader position. The optimiser state is typically twice the parameter count or more, so the checkpoint is several times the model size — hundreds of gigabytes to terabytes at frontier scale.

The costs:

• Time to write, during which training is stalled if the checkpoint is synchronous.
• Storage bandwidth, which at these sizes is a serious infrastructure requirement in its own right.
• Storage capacity, since several checkpoints must be retained.

The optimum frequency balances checkpoint cost against expected work lost, and the classical result applies: the interval should be roughly the square root of twice the checkpoint cost divided by the failure rate. More frequent failures justify more frequent checkpoints, and the arithmetic is worth doing rather than choosing a round number of hours.

Techniques that reduce the cost: asynchronous checkpointing, where training continues while the write proceeds from a snapshot; sharded checkpointing, where each rank writes only its own shard in parallel; in-memory checkpointing to a peer node's memory as a fast tier, with a slower durable copy taken less often.

Elastic training

Continuing with fewer nodes rather than halting requires the framework to reconfigure the collective communication and to adjust the effective batch size — which changes the optimisation trajectory and must be handled deliberately.

• Reconstitute the process group without restarting from scratch.
• Maintain the global batch size by adjusting gradient accumulation on the remaining nodes, so the training dynamics are unchanged.
• Hot spares in the cluster, so a failed node is replaced within minutes rather than the run being resized. For a large run, spare capacity is cheaper than the compute lost to restarts.

Straggler detection

A slow node is worse than a dead one. In synchronous training every step waits for the slowest rank, so a node running 20% slow makes the entire cluster 20% slow, indefinitely, with no error raised anywhere.

• Per-rank step-time monitoring, with automatic detection of persistent outliers.
• Automatic eviction and replacement of a straggler, treating it as a failure.
• Root causes are mundane and varied: thermal throttling, a degraded network link, a failing device, ECC error correction, or a noisy co-tenant. The detection matters more than the diagnosis, because the response is the same.

Scheduling

• Gang scheduling: a job runs when all its nodes are available, or not at all. Partial allocation is wasted allocation for synchronous training.
• Topology-aware placement, since collective communication performance depends heavily on network locality — a job spread across the wrong network boundary can be dramatically slower for no visible reason.
• Preemption with checkpointing for lower-priority jobs, so high-priority runs get capacity quickly.
• Queueing with fair share, so a large run cannot starve everything else indefinitely.

And the overall metric that matters: goodput — the fraction of wall-clock time spent making forward progress rather than restarting, re-computing lost work, waiting on stragglers, or checkpointing. Optimising anything else in isolation frequently reduces it.