A GPU platform has a few very large jobs competing with thousands of small ones. How should scheduling fairness, fragmentation, bin packing, preemption, queueing and starvation shape the architecture?

Why GPU scheduling is not CPU scheduling

GPUs are indivisible at the granularity most workloads need, expensive, and scarce. A CPU scheduler can timeshare; a GPU scheduler mostly cannot without partitioning support, and even then the partitions are coarse. This makes placement decisions costly and hard to reverse.

The fragmentation problem

A job needing eight co-located GPUs cannot run on a fleet with seven free per node, however much aggregate capacity is free. Fragmentation is the dominant inefficiency, and it is created by exactly the small jobs that are easiest to schedule.

The responses:

• Reserve node groups for large jobs, keeping whole nodes free even when small jobs are queued. This wastes capacity deliberately to preserve the ability to run large work at all.
• Bin-pack small jobs onto the same nodes aggressively, consolidating rather than spreading, so free capacity accumulates in contiguous blocks.
• Defragment by draining nodes, migrating small jobs when they are checkpointable. Only possible if the workload supports it, which is a constraint the platform can impose on the job format.

Starvation and fairness

Large jobs starve under naive fairness, because there is always a small job that fits and a large one that does not. The controls:

• Reservations that accumulate, so a large job's claim on resources grows as it waits and eventually blocks smaller jobs from taking the capacity it needs.
• Ageing priority, so waiting time increases scheduling priority.
• Separate queues with guaranteed shares, so large-job capacity is not a residual.

Preemption, and its real cost

Preempting a training job that has run for six hours without checkpointing destroys six hours of work and a large amount of money. So preemption is only viable where the platform requires or provides checkpointing, and the checkpoint interval becomes a scheduling parameter rather than a user choice.

A cheaper alternative that avoids most of this: offer explicit spot and on-demand classes, so preemption is a contract the customer accepted at a lower price rather than a decision the scheduler makes about work someone believed was safe.

The economic layer

Because the resource is expensive, utilisation is the business. A few percentage points of idle GPU is a large cost, which pushes toward aggressive packing — and aggressive packing increases the chance of interference between co-located jobs on memory bandwidth and interconnect. That interference is invisible to the scheduler and visible to the customer as inconsistent training throughput, which is why isolation guarantees, not just allocation, belong in the product definition.