A compute platform schedules GPU jobs with execution times ranging from minutes to weeks, where large jobs need many co-located accelerators. What happens if the scheduler uses simple first-come-first-served with best-fit packing?

What happens

Fragmentation, then starvation of exactly the jobs that matter most.

The mechanism is specific. A job requiring 64 co-located accelerators can only run when 64 are free on connected hardware. Under first-come-first-served with best-fit packing, small jobs continuously arrive and fill the gaps. Free capacity exists in aggregate — perhaps 30% of the fleet — but it is scattered as two here and four there, and never coalesces into a contiguous 64.

The large job waits indefinitely while utilisation reads healthy. The dashboard says the cluster is busy; the most valuable work in the queue has not started in three days.

Two further effects compound it:

• Partial allocation deadlock. If the scheduler grants accelerators as they free up, the large job holds 40 idle accelerators waiting for 24 more, while another large job holds 30 waiting for 34. Neither can proceed and both are consuming the resource.
• Locality ignored. Distributed training performance depends on interconnect topology. Sixty-four accelerators spread across the fleet may be functionally useless even when nominally allocated, because the collective communication becomes the bottleneck.

What the scheduler actually needs

1. Gang scheduling. All-or-nothing allocation: a job either receives its full topology-aware allocation or nothing at all. This alone eliminates the partial-allocation deadlock.

2. Reservations with backfill. Reserve a future slot for the large job, then allow small jobs to run in the gap only if they will finish before the reservation starts. This is the classic answer from high-performance computing, and it converts fragmentation from a starvation problem into a scheduling problem while keeping utilisation high.

3. Topology awareness. Allocate by interconnect locality, not by count. Placement is part of the allocation, not an afterthought.

4. Preemption with checkpointing. Low-priority long jobs can be suspended to make room, and resume from a checkpoint rather than from the beginning. Without checkpointing, preemption destroys work and becomes politically impossible to use.

5. Defragmentation. Periodically migrate small jobs to consolidate free capacity, the same problem as memory compaction and with the same solution.

6. Queue-time accountability. Fairness measured as wait time relative to job size, not as share of resources. Otherwise the metric rewards exactly the behaviour causing the starvation.

The trade-off to name

Every one of these reduces raw utilisation. A reservation leaves capacity deliberately idle. Gang scheduling refuses partial allocations that would have used free hardware. Defragmentation spends capacity on migration.

That is the correct trade, and it needs stating explicitly because utilisation is the metric executives watch: a cluster at 95% utilisation where large jobs never start is worth less than one at 80% where they do. The scarce resource is not the accelerator; it is the completed job.