A distributed training job runs at 90% scaling efficiency on 512 GPUs. At 1,024 GPUs it drops to 60%. Walk me through where the time went, what you would measure, and what you would try.

What the interviewer is testing

Whether you understand that a distributed training job is a communication problem in a compute costume, and whether you will insist on a measured breakdown before theorising about the network.

The clarifying questions that change the answer

• Which parallelism strategy — data, tensor, pipeline, or a combination? Each has a different communication pattern and hits a different wall, and the answer is meaningless without it.
• Did the global batch size change? Weak scaling (batch grows with device count) and strong scaling (batch fixed) degrade for different reasons, and "efficiency" means something different in each.
• What is the topology? High-bandwidth interconnect within a node and a slower fabric between nodes is the defining asymmetry of every large cluster.
• Is step time dominated by compute, by communication, or by data loading? One profile answers this and it is usually not the answer people expect.

Where the time goes

Gradient synchronisation. Data-parallel training performs an allreduce over the whole parameter set every step. Ring allreduce moves roughly 2(N−1)/N times the model size per device, which is nearly independent of N — but it completes in 2(N−1) sequential steps, so its latency grows linearly with participant count. At small scale the bandwidth term dominates; at large scale the latency term does, and the transition between 512 and 1,024 is exactly where that crossover typically lands.

Topology. Intra-node links are an order of magnitude faster than the inter-node fabric, which is why hierarchical collectives — reduce within a node, then across nodes — exist. Doubling device count usually doubles node count, so the share of traffic crossing the slow fabric rises even if nothing else changes.

The straggler effect. Every allreduce is a barrier, so the step costs as much as the slowest participant. With 1,024 participants you sample twice as deeply into the tail of the per-device step-time distribution, so the expected maximum rises even when every device is identical. This is tail amplification applied to compute, and it is the term people forget.

Everything that is not the accelerator: data loading, per-step host overhead, and checkpointing, which at this scale is a synchronised write of hundreds of gigabytes.

What I would measure, in order

1. Step time split into compute, communication and wait, from the framework profiler.
2. Collective time per step and its variance, plus the algorithm chosen and the bandwidth achieved, from the collective library's own instrumentation.
3. Per-rank step-time distribution, to separate "one bad node" from "the distribution widened".
4. Placement: are communicating groups co-located on a node, or scattered across the fabric.

What I would try

Overlap communication with the backward pass by bucketing gradients so reduction starts before backward finishes · hierarchical collectives that keep most traffic intra-node · reduced precision or compression for the reduction · a larger per-device batch to raise the compute-to-communication ratio · and, past a certain scale, a different decomposition entirely, because pure data parallelism stops being the right answer when the allreduce latency term dominates.

Common weak answers

• "Add more GPUs." Scaling efficiency is the measurement that says adding more will make it worse.
• "It is the network, buy a faster fabric." Possibly true, unfalsifiable without the breakdown, and expensive to be wrong about.

What a strong answer adds

Scaling efficiency is an economic number, not a technical one. Sixty percent at 1,024 devices may be the correct choice when wall-clock time to a result is worth more than the wasted capacity, and that trade belongs to whoever is paying rather than to the engineer. Saying so is what separates an answer about a profiler from an answer about a decision — and the same reasoning governs large-scale inference, where the collectives are smaller and the latency budget is far tighter.