Preemption, Priority and Spot Capacity
How interruptible capacity changes the cost of training by a large factor, what a job must be able to do to use it, and the failure modes that make spot capacity a false economy for the unprepared.
Interruptible cloud capacity is typically offered at a substantial discount to on-demand pricing, often in the range of 60 to 90 percent depending on instance type and region, in exchange for the provider's right to reclaim it with a short notice period. For a workload that can tolerate interruption, that is the single largest cost lever available in ML infrastructure. For one that cannot, it is a way to lose a week of training.
What a job needs to survive preemption
Frequent, fast checkpointing. The work lost on preemption is everything since the last checkpoint, so checkpoint interval is a direct tradeoff against expected loss. The interval that minimises total expected cost depends on checkpoint cost and preemption rate, and the classic result is that the optimum is roughly proportional to the square root of the product of checkpoint time and mean time between interruptions.
Fast restart. Detecting the loss, reacquiring capacity, reloading state and resuming must take minutes rather than hours, or the recovery cost exceeds the saving. Keeping the container image cached and the checkpoint in a nearby, fast store are the practical requirements.
Graceful handling of the notice. Providers give a short warning before reclamation, typically on the order of a couple of minutes. A job that catches that signal and writes a checkpoint immediately loses almost nothing; one that ignores it loses everything since its last scheduled checkpoint.
Tolerance of changing world size, if the job is to continue with fewer workers rather than waiting for replacements. This is elastic training, and it interacts with batch size and learning rate as noted elsewhere.
Where it does and does not fit
Pretraining and long fine-tuning runs fit well: they are long, checkpointable, and not latency-sensitive. Hyperparameter search fits especially well, since individual trials are independent and losing one costs little. Batch inference fits.
Online serving does not. Losing capacity under load is a availability incident, and the saving is not worth it for the tier that users depend on. The common architecture is on-demand or reserved capacity for the baseline serving tier and interruptible capacity for training, batch work and burst overflow behind a queue.
When it breaks
Preemption rates are not stationary. Interruptible capacity for a popular GPU type in a busy region can go from rarely reclaimed to reclaimed within minutes, with no warning and no SLA. A pipeline that works for months can become unusable in a week, which means the fallback to on-demand has to be automatic rather than manual.
Checkpoint writes are expensive at scale. A large model's optimiser state can be several times the parameter memory, so writing a checkpoint moves a great deal of data and stalls training while it happens. Asynchronous and sharded checkpointing exist precisely because the naive version makes frequent checkpointing unaffordable.
Correlated preemption removes everything at once. Reclamation is often driven by capacity pressure on an instance type in a zone, so a job's workers are not preempted independently. Diversifying across instance types and zones reduces correlation and complicates the placement requirements that gang scheduling already imposes.
The saving is smaller than the headline. Restart overhead, lost work, engineering effort and the on-demand fallback all subtract from the discount. The honest comparison is total cost per completed training run, not price per GPU-hour, and for short jobs with expensive checkpoints the arithmetic can come out negative.
12 flashcards for this concept
Click a card to reveal the answer.