An AI platform must secure accelerator capacity where supply is constrained and demand is uncertain. How should reservations, on-demand and spot be combined?

Why this differs from ordinary capacity purchasing

For most workloads, on-demand capacity is always available at a price, so commitments are purely a cost optimisation. For scarce accelerator generations, on-demand capacity may simply not exist — the reservation is not a discount, it is the only way to have the resource at all.

That changes the analysis fundamentally. The cost of under-reserving is not a higher unit price; it is being unable to run the work.

The portfolio

1. Reserved capacity for the committed floor. Sized to the demand you are confident about — production inference baseline and ongoing training programmes. Long commitments, best price, and guaranteed availability, which is the property that actually matters.

2. On-demand for the uncertain middle, where it exists. Accept the premium for flexibility, and do not assume availability during broad demand events.

3. Spot and preemptible for interruptible work. Experimentation, evaluation runs, batch inference and any training that checkpoints frequently. Substantial discounts in exchange for reclamation risk, which is acceptable precisely when restart is cheap.

4. Multi-provider allocation as a capacity strategy, not an anti-lock-in one. When the binding constraint is physical supply allocated quarters in advance, a second provider is the only way to obtain more. This is the one genuinely strong multi-cloud justification: name the constraint, name the date it binds, and name what you cannot do without the second provider.

The architectural requirements the portfolio imposes

• Checkpointing everywhere, so any job can be preempted and resumed. Without it, spot and preemption are unusable and the cheapest tier is unavailable.
• Workload placement flexibility, so jobs can run wherever capacity exists — which constrains how deeply you can use provider-specific services on that path.
• Priority classes and preemption, so scarce reserved capacity is used by the highest-value work and lower-priority jobs act as a buffer that yields under pressure.
• Gang scheduling and topology awareness, since fragmented capacity is not usable capacity for large jobs, and a cluster can report high utilisation while the most valuable work never starts.

The forecasting discipline

Commit against a distribution, not a point forecast, and make the asymmetric cost of error explicit: under-reserving costs delayed work and possibly unusable capacity elsewhere; over-reserving costs idle hardware on a depreciation schedule. Those asymmetries are rarely equal, and the target should follow the asymmetry rather than a utilisation number.

The reporting note

Report queue time by job size, not utilisation. Utilisation is what leadership asks for and it is actively misleading here — the scarce resource is the completed job, not the occupied accelerator.