Spot Interruption Budget

Interruptible capacity is offered at 60–90% below on-demand, and the discount is quoted per node-hour. The cost that is not quoted is the work destroyed when a node is taken back, and whether the trade is good depends entirely on a property of the workload rather than of the price.

An interruption budget makes that explicit: expected interruptions per hour for the whole workload, multiplied by the work lost per interruption, compared against the discount. The arithmetic is short and it changes decisions.

Why it matters

Because the per-node interruption rate is the wrong number, and it is the one everybody uses. Rates add across the gang. At 5% per node per hour, a single node fails roughly every 20 hours; a 256-node synchronous job sees 256 x 0.05 = 12.8 reclamations an hour, a mean time between failures of about 4.7 minutes. If a checkpoint takes 8 minutes, the job cannot complete one between failures and makes almost no progress at any price.

The same calculation rules interruptible capacity in for the opposite shape of workload. For independent tasks - batch inference, hyperparameter search, CI jobs, data preparation - a reclamation costs one task, the rate multiplication never happens, and the discount lands almost in full.

Implementation patterns

• Compute the workload MTBF, not the node MTBF: per-node rate times node count, inverted.
• Size the checkpoint interval from the standard optimum (Young, 1974; Daly, 2006): interval ≈ sqrt(2 x checkpoint cost x MTBF). If the result is close to the checkpoint cost itself, the workload does not belong on interruptible capacity.
• Attack the checkpoint cost, which is the lever with two orders of magnitude in it. Write to local NVMe, flush asynchronously to object storage, and keep a replica of each rank's state in a peer's memory, so a checkpoint is seconds rather than minutes.
• Use the reclamation notice. Providers give short warning - two minutes is the common figure - which is enough to flush a checkpoint, drain a queue consumer or deregister from a load balancer, but only if something is listening for it.
• Mix the fleet by role: on-demand or reserved capacity for ranks holding state and for the scheduler, interruptible capacity for elastic workers, with a framework that can continue at reduced width.
• Diversify instance shapes and zones, because reclamations inside one capacity pool are correlated and the independence assumption in the arithmetic is optimistic.

Industry example

The split is visible in how large accelerator fleets are operated: elastic and preemptible pools carry evaluation, data processing and search workloads, while long synchronous training runs sit on committed capacity with fast local checkpointing. The published checkpointing literature from high-performance computing is the same body of work - the optimal-interval result dates from 1974 and is still the right first calculation - and the modern refinement is in-memory or peer-replicated checkpoints, which exist precisely because the interval formula is dominated by checkpoint cost.

Failure scenarios

• A gang-scheduled job on interruptible capacity that never finishes, burning the discount in restarts while appearing to be making progress.
• Correlated reclamation takes a whole pool at once, so the replacement capacity is unavailable for minutes and the job's restart queues behind everyone else's.
• No handler for the termination notice, so in-flight requests are dropped and queue messages are redelivered as duplicates.
• Checkpoint to shared object storage from every rank simultaneously, which saturates the path and makes the checkpoint slower as the job grows.
• A stateful service quietly placed on interruptible nodes by a cluster autoscaler default, discovered during the first reclamation.

Trade-offs

What is gained is a 60–90% discount on the compute line. What is paid is engineering: checkpoint machinery, a notice handler, replacement logic, elastic width, and a scheduler that understands two capacity classes. For a job that runs for two hours once a week, that engineering costs more than the discount recovers for years. The second cost is schedule risk - interruptible capacity can be unavailable, not merely interrupted - which makes it unsuitable for anything with a deadline the business cares about.

When not to use it

Do not put a synchronous job on interruptible capacity when the workload MTBF is within a small multiple of the checkpoint cost, and do not put anything on it that holds unique state or serves user traffic with a tight latency target. Avoid it also for short jobs, where the fixed engineering cost dominates, and for anything whose completion time is promised to someone. The clean decision rule: interruptible capacity is cheap when the unit of loss is one task, and expensive when the unit of loss is the whole job.

Interview question

Q: A team wants to move a 128-GPU training job onto interruptible capacity for the 70% discount. Reclamations run at about 4% per node per hour and a checkpoint takes 6 minutes. What do you tell them?

What a strong answer covers: computing the job MTBF as 1/(128 x 0.04) hours, about 12 minutes, and the optimal interval as sqrt(2 x 6 x 12) ≈ 12 minutes, so roughly half the time is checkpoint and re-done work; identifying checkpoint cost as the lever and naming local NVMe plus async upload and peer-memory replicas; proposing a mixed fleet or elastic width as the alternative; noting that correlated reclamation makes the estimate optimistic; and being willing to recommend on-demand for a job that is short or deadline-bound.

Quick check

Quiz: Why does a 256-node job fail every few minutes on capacity where a single node lasts 20 hours? Interruption rates add across the gang: 256 x 5% per hour is 12.8 an hour, an MTBF of about 4.7 minutes.

Flashcard: Optimal checkpoint interval? Roughly sqrt(2 x checkpoint cost x MTBF) - and if that is near the checkpoint cost itself, the workload does not belong on interruptible capacity.