A nightly batch job takes six hours on on-demand instances. Someone proposes spot to save 70%. What must be true?

What the interviewer is testing

Whether you can assess interruption tolerance concretely rather than treating spot as a discount.

What must be true

The work must be checkpointable or retryable. A six-hour job that restarts from zero on interruption may never complete — if the expected time between interruptions is shorter than the runtime, it makes no progress at all. Decompose it into tasks of tens of minutes with durable progress, so an interruption loses one task rather than the whole run.

The interruption notice must be handled. A two-minute warning is enough to checkpoint, drain and deregister — but only if a handler exists. Without one, the difference between graceful and abrupt loss is entirely lost.

The deadline must have slack. If the job must complete before the business day starts, and interruptions can extend it, there must be either enough margin or a fallback to on-demand as the deadline approaches.

Capacity must be diversified across instance types and availability zones. Interruption correlates within a pool, so a fleet drawn from one type can be reclaimed together — which is the failure that turns an occasional interruption into a total loss.

The orchestrator must reschedule interrupted work automatically.

The design that works

A mixed fleet: a baseline of on-demand or reserved capacity guaranteeing progress, topped up with spot for the bulk. A large-scale reclamation then degrades throughput rather than stopping the job.

Plus an escalation rule: if the deadline is at risk, shift to on-demand for the remainder. The saving is on the typical night, not on every night.

The framing for the business

Spot is not a discount on the same product — it is a different service level. Buying it for a workload that cannot absorb interruption converts a saving into an incident, and that distinction should be stated plainly when the proposal is made.

What a strong answer adds

Checking whether the job needs to be six hours at all. A poorly parallelised batch job is often the larger saving, and shortening the runtime also reduces interruption exposure.

Common weak answers

Approving on the discount. Rejecting spot without assessing checkpointability.