Which workloads genuinely suit spot or preemptible capacity, what must be true of them, and what are the failure modes of over-applying it?

What spot capacity actually is

Unused provider capacity offered at a large discount — commonly 60–90% — with the condition that it can be reclaimed at short notice, typically a couple of minutes' warning or less.

The discount is payment for accepting interruption. Everything about suitability follows from whether the workload can genuinely tolerate that.

Genuinely suitable

• Batch processing with checkpointing. Interruption costs the work since the last checkpoint, and the job resumes elsewhere.
• CI and build workloads, where a failed job is simply re-run and the delay is minor.
• Stateless serving with enough replicas and fast startup, where losing a fraction of capacity is absorbed by the remainder — provided the replacement can start faster than further reclamations occur.
• Data processing frameworks with built-in resilience, which re-execute lost tasks by design.
• Model training with periodic checkpoints, where the checkpoint interval sets the maximum loss.
• Development and test environments, where interruption is an inconvenience.

Not suitable

• Stateful services without fast failover: databases, brokers, anything holding authoritative state where reclamation means a failover event.
• Long single operations with no checkpoint, where interruption discards all progress — and where enough interruptions mean the job never completes at all, which is the failure mode people do not anticipate.
• Latency-critical serving with no headroom, where losing capacity immediately degrades users.
• Workloads whose startup takes longer than the reclamation warning, which cannot drain gracefully.
• Anything where reclamation cascades — losing capacity increases load on the remainder, causing failures that look like a capacity incident.

What must be true

• Graceful handling of the termination signal: drain connections, checkpoint, deregister — within the warning window, which must be measured rather than assumed.
• Diversification across instance types and zones, since reclamation is often correlated within a type. A fleet on one spot instance type can lose all of it simultaneously.
• A fallback to on-demand capacity, automatic, so a spot shortage degrades cost efficiency rather than availability.
• A baseline on committed capacity, with spot used for the variable portion — not the whole fleet.
• Interruption rate monitored, because it varies by type, zone and time and a workload that was suitable can become unsuitable without any change on your side.

The failure modes of over-applying it

• Reclamation cascades, where the fallback is slower than the reclamation rate and capacity ratchets down.
• Jobs that never complete, repeatedly interrupted before reaching a checkpoint — so the checkpoint interval must be shorter than the expected time between interruptions, which is a real calculation.
• Hidden statefulness discovered during reclamation, where a "stateless" service turns out to hold session or cache state that mattered.
• Savings that evaporate into engineering time spent managing interruptions, which is a genuine cost and is rarely counted.
• Correlated reclamation during a demand surge, when the provider reclaims capacity precisely because on-demand demand is high — which is likely to coincide with your own peak. This is the most important and least appreciated risk: spot capacity is least available exactly when it is most needed.