A recommendation platform runs large offline training and feature computation jobs. What must be true architecturally before spot capacity is usable, and what does it cost?

What must be true

1. Work is checkpointable and restartable. Reclamation can occur at any moment with short notice. A job that must restart from the beginning after two hours of work is not a spot workload — the expected cost of lost work can exceed the discount.

2. Restart is idempotent. A job that partially completed and restarts must not double-count, double-write or corrupt its output. This usually means writing to a staging location and committing atomically on completion.

3. Reclamation is handled gracefully. The termination notice, though short, is enough to checkpoint and deregister if the job is built for it.

4. The workload has slack. A job that must finish by a deadline cannot depend entirely on capacity that may disappear. Mixed placement — a reserved core with spot for the tail — is the usual resolution.

5. Capacity is fungible across instance types and zones. Spot availability varies by type and location, so a job pinned to one configuration will be starved. Diversification across many pools is the single largest determinant of effective spot availability.

What it costs

• Engineering effort in checkpointing, which is real work and adds complexity to every job.
• Wasted computation between the last checkpoint and reclamation, which sets the checkpoint frequency — more frequent checkpoints reduce waste and add overhead.
• Scheduling complexity, since the scheduler must handle disappearing capacity, replacement and rescheduling.
• Unpredictable completion times, which propagates into any downstream dependency on the output.
• Correlated reclamation. Spot capacity is reclaimed when demand rises, so many instances can disappear together — which means the design must survive losing a large fraction at once, not just one instance.

That last point is the one teams underestimate: spot is not independent random failure, it is correlated withdrawal driven by market demand.

Where it fits and where it does not

Fits: model training with checkpoints, feature backfills, batch inference, evaluation runs, data processing, CI workloads.

Does not fit: online serving, stateful services holding connections, anything with a hard deadline and no slack, and — importantly — anything whose restart is expensive relative to its runtime.

The organisational benefit

Spot forces checkpointing and idempotent restart, which are properties worth having regardless: they make jobs resilient to ordinary failures, enable preemption of low-priority work for high-priority work, and make capacity fungible. Teams that adopt spot for cost reasons frequently find the reliability improvements more valuable than the discount.