A Kubernetes platform shows GPU utilisation below 40% because jobs request whole nodes. How should bin packing, gang scheduling, preemption, partitioning and queue-based admission raise utilisation without harming priority jobs?

Diagnose the cause first, because the remedies differ entirely

Low GPU utilisation has several distinct causes and they are frequently confused:

• Allocation waste — a job holds eight GPUs and uses two. The scheduler sees the cluster as full while the hardware is idle, and no scheduling change fixes this; the request must change.
• Idle allocation — notebooks and development sessions holding GPUs for hours while someone thinks. Usually a large fraction of the loss and the easiest to address.
• Fragmentation — free GPUs scattered across nodes such that no node has enough for the next job.
• Pipeline stalls — data loading, checkpointing or synchronisation leaving the GPU waiting. This is utilisation lost inside an allocated job, which cluster scheduling cannot help at all.
• Queue waiting, where jobs are ready and the scheduler cannot place them.

Measure GPU utilisation inside allocations separately from allocation rate, because the two numbers point at completely different interventions and reporting only one is how teams optimise the wrong thing.

The mechanisms

• GPU partitioning (MIG-style), splitting a physical accelerator into isolated instances. This is the single highest-value change for inference and development workloads, which frequently need a fraction of a GPU and currently take a whole one.
• Time-slicing for development and low-priority workloads where isolation is less critical, allowing oversubscription.
• Bin packing rather than spread scheduling, consolidating jobs onto fewer nodes so that whole nodes remain free for large jobs. Spread scheduling — the default in many configurations — actively creates the fragmentation problem.
• Gang scheduling: a distributed job starts when all its resources are available or not at all. Partial allocation is wasted allocation for synchronous training, since the job cannot progress and is holding GPUs others could use.
• Preemption with checkpointing, so low-priority work yields to high-priority work. This is what makes filling the cluster with opportunistic work safe — without it, backfill jobs block priority jobs and the utilisation gain costs more than it delivers.
• Queue-based admission with fair share, so a large job cannot starve everything else and a single team cannot monopolise the cluster.
• Topology-aware placement, since collective communication performance depends heavily on network locality — a job spread across the wrong network boundary is dramatically slower for no visible reason, which converts a utilisation problem into a goodput problem.
• Idle detection and reclamation for interactive sessions, with warning.

Protecting priority jobs while raising utilisation

The combination that works:

Reserved capacity for high-priority work + opportunistic backfill with preemption + checkpointing in the backfill jobs. Priority jobs schedule immediately against the reservation; the remainder of the cluster runs low-priority work that yields within minutes when a priority job needs the space.

The precondition is that backfill jobs checkpoint, or preemption discards hours of work and teams resist it politically until the mechanism is disabled.

The organisational half

• Quotas per team with a burst allowance, since weighted fair queueing over available capacity beats hard caps — a team can exceed its share when the cluster is idle and is constrained only under contention.
• Cost visibility per team, because an idle GPU that nobody is charged for is an idle GPU forever.
• Right-sized defaults in the job templates, since most over-requests are the template's fault rather than a deliberate choice.
• An easy way to request a fraction of a GPU, or everyone will request a whole one.

The largest single win in most clusters is not scheduling sophistication — it is stopping people from holding whole GPUs for interactive work they are not currently using, and that is a defaults-and-reclamation problem rather than an algorithmic one.