Fragmentation Cost

A fleet with plenty of free capacity can be unable to run a job. A workload needing eight co-located accelerators cannot run on a cluster with seven free on every node, no matter how much total capacity is idle.

Fragmentation is capacity that exists and cannot be allocated, and in scheduling indivisible expensive resources it is usually a larger loss than genuine over-provisioning.

Why it matters

It is created by the jobs that are easiest to schedule. Small jobs fit anywhere, so a naive scheduler spreads them, and spreading is exactly what destroys contiguous capacity. The scheduler's default behaviour causes the problem it will later be blamed for.

It also causes starvation: large jobs wait indefinitely because there is always a small job that fits and a large one that does not.

Implementation patterns

• Bin-pack rather than spread. Consolidate small jobs onto the fewest nodes so free capacity accumulates in contiguous blocks. This is the opposite of the spread-for-resilience instinct and it is correct here.
• Reserve node groups for large jobs, keeping whole nodes free even while small jobs queue. Deliberately wasting capacity to preserve the ability to run large work at all.
• Accumulating reservations, so a waiting large job progressively blocks smaller ones from taking the resources it needs. Without this, large jobs starve.
• Ageing priority, so waiting time raises scheduling priority.
• Defragment by draining and migrating checkpointable jobs — only possible if the platform requires checkpointing, which is a constraint worth imposing on the job format.
• Separate queues with guaranteed shares, so large-job capacity is a commitment rather than a residual.

Industry example

GPU platforms such as RunPod, Modal and Together AI face this constantly, because the job-size distribution is extremely wide: thousands of single-accelerator inference workloads alongside multi-node training runs. The economic pressure makes it acute — utilisation is the business, so a few points of unusable capacity is a large cost.

The same structure appears in device-testing platforms, where a suite needing a specific device configuration cannot use free capacity of another type, and in any batch platform with heterogeneous resource requests.

Failure scenarios

• Spread scheduling, which maximises fragmentation.
• Large jobs starving with no reservation or ageing mechanism.
• Preemption without checkpointing, which destroys hours of work and money to recover capacity.
• Aggressive packing causing interference on memory bandwidth or interconnect, invisible to the scheduler and visible to the customer as inconsistent throughput.
• Fragmentation not measured, so the platform appears to have capacity and cannot serve requests, with no metric explaining the contradiction.

Trade-offs

Packing tightly maximises allocatable capacity and increases co-location interference. Reserving for large jobs guarantees they can run and wastes capacity while they are absent. Preemption recovers capacity and destroys work.

There is no configuration that is right for all job mixes, which is why mature platforms expose the trade to the customer as priced classes — on-demand versus preemptible, dedicated versus shared — so preemption becomes a contract accepted at a lower price rather than a scheduler decision about work someone believed was safe.

Interview question

"Your cluster reports 30% of accelerators free and a large training job has been queued for six hours. Explain what is happening, and give me two changes with different trade-offs."