A 256-GPU synchronous training job is proposed for interruptible capacity at a 70% discount. Nodes are reclaimed at roughly 5% per node per hour and a full checkpoint takes 8 minutes. Estimate whether the discount survives the interruptions.

The assumptions, stated

Synchronous data-parallel training: all 256 ranks must be present for a step, so losing one rank stops the job. Reclamation is independent per node at 5% per hour. A checkpoint costs 8 minutes of wall time, and after a reclamation the job restarts from the last checkpoint once a replacement is placed.

The arithmetic

Per node, mean time between reclamations is 1 / 0.05 = 20 hours. For the job, the rate is what matters and rates add: 256 nodes x 0.05 per hour = 12.8 reclamations per hour, so the job's mean time between failures is about 4.7 minutes.

That number is the whole answer, and it is the thing teams get wrong: an interruption rate that is benign per node is multiplied by the size of the gang. A 4-node job on the same capacity sees a failure every 5 hours; a 256-node job sees one before a single checkpoint finishes.

Apply the standard optimum (Young, 1974; Daly, 2006): the best checkpoint interval is roughly the square root of twice the checkpoint cost times the MTBF, so sqrt(2 x 8 min x 4.7 min) ≈ 8.7 minutes. The interval is barely longer than the checkpoint itself, which means the job spends roughly half its time writing checkpoints and the other half re-doing lost work. Expected useful throughput is a small fraction of nominal, and the 70% discount is comfortably erased.

Which assumption dominates the error

The checkpoint cost, because it is the one you can change by two orders of magnitude. Write to local NVMe with an asynchronous flush to object storage, keep a replica of each rank's state in the memory of a peer, and a checkpoint drops from 8 minutes to seconds. At C = 20 seconds, the optimum interval is sqrt(2 x 20 s x 282 s) ≈ 106 seconds and overhead falls to roughly a fifth. Still substantial, now arguably worth 70%.

The second-order assumption is independence. Reclamations in the same capacity pool are often correlated, which makes the arithmetic above optimistic rather than pessimistic.

What the number rules in or out

• It rules out interruptible capacity for large gang-scheduled jobs with slow checkpoints, which is the configuration most often proposed because the discount looks irresistible on a spreadsheet.
• It rules in interruptible capacity for embarrassingly parallel work: batch inference, hyperparameter search, data preparation, CI. Losing one worker costs one task, not the job, so the MTBF multiplication never happens.
• It rules in a mixed fleet: on-demand or reserved capacity for the ranks that hold state and interruptible capacity for elastic workers, with a framework that can continue at reduced width rather than stopping.
• The general decision rule: interruptible capacity is cheap when the unit of loss is one task and expensive when the unit of loss is the whole job. It flips as soon as the restart cost falls below a few per cent of the interval between interruptions.

When not to bother with interruptible capacity

For a job that takes two hours and runs once a week, on-demand costs a few hundred pounds and needs no checkpointing machinery at all. Engineering time spent making a small job interruption-tolerant is almost always worth more than the discount it recovers.