A training pipeline runs on 400 interruptible GPU instances checkpointing every 30 minutes. Regional demand spikes and reclamation goes from a handful of instances an hour to most of the fleet inside ten minutes. What happens minute by minute, and what stops it?

Minute by minute

Minute 0. Reclamation notices arrive in bulk. Providers give short warning — on the order of two minutes for spot instances on one major cloud and tens of seconds for some preemptible families, as published in 2026 — so the fleet has one window, not a queue of them.

Minute 0 to 2. Every worker reacts identically: flush a checkpoint. Hundreds of multi-gigabyte writes hit the same object-storage bucket within seconds of each other. Requests to a single prefix get throttled, writes slow, and a fraction of checkpoints do not complete before the instance is taken. Those jobs lose everything since their last successful checkpoint, up to the full 30-minute interval.

Minute 2 to 5. The scheduler sees hundreds of unschedulable jobs and re-requests capacity from the pool that just evicted them. The capacity API returns nothing, repeatedly, from every worker at once.

Minute 5 onward. If on-demand fallback is configured without a ceiling, it succeeds. Hourly cost for the fleet jumps by roughly 3–4×, silently, and the first person to notice is whoever reads the bill next month. If fallback is not configured, the pipeline simply stops making progress while appearing healthy: jobs exist, queues are non-empty, nothing errors.

Where it amplifies

Two feedback loops. The synchronised checkpoint storm turns a capacity event into a data-loss event by saturating the one shared resource every worker needs at once. The retry loop turns a shortage into sustained load on the capacity API, which does not create capacity.

What the user sees

Nothing, for hours. Then a training run that was due Friday lands Tuesday, or a cost anomaly alert fires on a number that is already spent.

What stops it

• Capacity diversification: draw from several instance families and zones, so a pool exhaustion is a partial event. A single-family fleet has one failure domain by construction.
• Staggered checkpoint offsets per worker, so the storm is spread across the interval rather than aligned. This costs nothing and removes the amplifier.
• A spend ceiling on fallback, expressed in currency per hour, not in instance count. An instance cap does not bound cost when the instance price differs by 4×.
• A checkpoint interval derived from measured mean time between reclamations, not chosen round. The rule: expected lost work per interruption is half the interval, so if reclamations arrive faster than about twice the checkpoint interval, the job's expected progress per wall-clock hour goes to zero and it never finishes. That is the cliff, and it is arithmetic, not bad luck.

What would have to be true for it to self-heal

Work must be idempotent and resumable at chunk granularity rather than epoch granularity, capacity must be sourced from pools that are not correlated, and the scheduler must back off exponentially with jitter instead of polling. With all three, the pipeline degrades to slow. With none, it degrades to stopped while looking busy.

When this is the wrong answer

For a job that completes in under an hour and can simply be rerun, all of this machinery is waste. Rerun it and take the discount.