A platform's accelerator fleet reports high utilisation, yet throughput per accelerator is far below the hardware's capability. Where does the waste usually hide?

Why utilisation is a misleading signal

Utilisation typically measures whether the device is busy, not whether it is doing useful work efficiently. A device can report high utilisation while executing small, inefficient operations, waiting on memory, or processing padding.

Where the waste hides

1. Input pipeline starvation. The accelerator waits for data. If loading, decoding and augmenting cannot keep up, the expensive resource idles between batches while reporting busy. This is the most common and most overlooked cause, and it is fixed with cheaper resources — more loader workers, prefetching, better storage throughput, caching decoded data.

2. Small batch sizes. Throughput rises with batch size until memory limits. Running below the memory ceiling leaves substantial throughput unused, and the reason is often a default nobody revisited.

3. Padding waste. Batching variable-length inputs by padding to the longest member means computing on padding. Length-grouped batching, or continuous batching where finished sequences leave and new ones join, recovers it — and for highly variable lengths this is a large effect.

4. Precision left on the table. Running at higher precision than the workload requires, when lower precision would deliver equivalent quality at substantially higher throughput.

5. Synchronisation and communication overhead in distributed work, where devices wait at collective operations. If the interconnect topology does not match the parallelism strategy, communication dominates and adding devices makes it worse.

6. Fragmentation. Free capacity exists in aggregate but is scattered, so large jobs cannot start. Utilisation looks excellent while the most valuable work waits — the metric that looks healthiest coinciding with the worst outcome.

The measurement that finds it

Throughput per device against a known achievable benchmark, not utilisation. Then profile to attribute the gap: time waiting on input, time in communication, time in computation, and computation efficiency against theoretical peak.

Alongside it, queue time by job size rather than aggregate utilisation, because that is what reveals fragmentation.

The economic framing

Accelerators are the most expensive resource in the system, so a percentage improvement in efficiency is worth more than the same percentage anywhere else. That justifies engineering effort — profiling, pipeline optimisation, batching strategy — that would be unwarranted for cheaper resources.

The corollary is that the cheap resources feeding them should be over-provisioned deliberately. Starving a very expensive device to save on storage throughput or loader CPU is a false economy, and it is a common one because those budgets are owned by different teams.