A 64-GPU distributed training job's step time rises from 310 ms to 440 ms after one failed node is replaced. GPU utilisation reads 96% on every node and nothing in the model or data pipeline changed. Where is the time going?

The first three things to look at, in order

1. The collective's bandwidth per rank, not the GPU's utilisation. Step time in data-parallel training is compute plus the all-reduce of gradients. If compute is unchanged, the 130 ms is in the collective.
2. The topology the new node was given. Within a node, GPUs communicate over a high-bandwidth fabric (NVLink and NVSwitch on NVIDIA systems, hundreds of gigabytes per second); between nodes they communicate over the network at a fraction of that. The replacement node may be in a different rack, behind a different leaf switch, or attached to fewer network interfaces than its predecessor.
3. Which network interface each rank is actually using. NCCL selects interfaces and builds its rings and trees from the topology it discovers at initialisation. A node whose GPU-to-NIC affinity differs - the NIC on a different PCIe root complex from the GPU it serves - forces traffic across the inter-socket link, and a ring that crosses that link once pays for it on every step.

The diagnosis

A collective is as fast as its slowest link, and the ring passes through every rank. A ring all-reduce moves roughly 2(N−1)/N times the gradient size per rank, so for a 1 GB gradient at 64 ranks each rank sends and receives close to 2 GB per step. At an effective 100 GB/s that is about 20 ms; at an effective 10 GB/s it is about 200 ms. One badly placed rank does not slow itself down, it slows the ring, and every other rank waits. That is why all 64 nodes show high utilisation: they are busy waiting inside the collective, which the GPU counts as occupancy.

The misleading signal is exactly that 96%. GPU utilisation measures whether a kernel is resident, not whether it is doing arithmetic, and a blocked NCCL kernel is resident. Teams chase the data loader for days on the strength of that number. The counters that would settle it in minutes are the achieved bus bandwidth of the collective and the per-rank wait time, both of which NCCL can report.

The fix

Confirm first with a standalone collective benchmark across the same 64 ranks, then with the new node excluded. If excluding it restores 310 ms, the cause is placement, and the fixes are scheduling-level rather than code-level: require the scheduler to place a job within one network domain and refuse to backfill a replacement rank outside it, pin interfaces explicitly rather than relying on discovery, and verify GPU-to-NIC affinity in the node's acceptance test before it rejoins the pool.

The alert that would have caught it earlier

Achieved collective bandwidth per job, with a floor. A job whose all-reduce bandwidth drops below, say, 70% of the cluster's known-good figure is reported as degraded even though it is making progress. The related change is to capacity reporting: free GPU count is not capacity for a 64-GPU job unless those GPUs are co-located, so the useful metric is the largest topologically contiguous block available, not the sum of free devices.

When this is the wrong answer

On a job small enough to fit inside one node, there is no inter-node collective and none of this applies: an unexplained step-time rise there is thermal throttling, a noisy neighbour on the same host, or the data pipeline, and the profiler is the right tool. Topology-aware scheduling is also the wrong first investment below roughly a few dozen nodes - it constrains the scheduler, lowers cluster utilisation and strands capacity, which is a real cost paid every day against a fault that happens occasionally. Prefer the cheap version until the fleet is large enough to justify it: an acceptance test that measures a node's achieved collective bandwidth before it rejoins the pool.