A team wants to data-parallel train a 70-billion-parameter model across 32 DGX H100 nodes and asks whether a single step is communication-bound. Using the vendor figures - 900 GB/s of NVLink bandwidth per H100 and eight 400 Gb/s adapters per node - roughly how long does one gradient all-reduce take flat across the cluster versus hierarchically, and which assumption dominates the error?

The assumptions, stated

Four numbers, each labelled with where it comes from:

• Gradient volume. 70e9 parameters at 2 bytes each in bf16 = 140 GB of gradients per step. This is arithmetic, not an estimate.
• Collective volume. A ring all-reduce moves 2(n-1)/n × D bytes per rank. NVIDIA's own nccl-tests documentation uses exactly that factor to convert algorithm bandwidth into bus bandwidth. For large n the factor is ~2, so each rank pushes about 280 GB per step.
• Intra-node bandwidth. NVIDIA's Hopper architecture material (2022) gives the H100 eighteen fourth-generation NVLink links totalling 900 GB/s per GPU.
• Inter-node bandwidth. The DGX H100 datasheet lists eight ConnectX-7 adapters at up to 400 Gb/s for the compute fabric. That is 3.2 Tb/s = 400 GB/s per node, shared by 8 GPUs, so 50 GB/s per GPU - about 18x narrower than the NVLink path.

The arithmetic

Flat ring across all 256 GPUs. Every hop crosses the narrow fabric, so the whole 280 GB per rank is priced at 50 GB/s: 280 / 50 = 5.6 seconds of pure communication per step.

Hierarchical. Reduce-scatter inside the node over NVLink, then all-reduce only the reduced slice across nodes, then all-gather back inside the node:

• Inside the node: 280 GB at 900 GB/s = 0.31 s
• Across nodes: each GPU now owns 140/8 = 17.5 GB, so 2 x 17.5 = 35 GB at 50 GB/s = 0.70 s
• Back inside the node: 0.31 s
• Total: roughly 1.3 s, or about 4x better than flat

The number, with its range. Flat 5.6 s against hierarchical 1.3 s, both at nominal link rates, so both are floors. Treat the real figures as 1.5x to 3x those.

Which assumption dominates the error

Not the gradient volume and not the link ratings. It is the achieved fraction of nominal bandwidth, which depends on message size, the number of channels, the ring the topology actually builds, and whether other traffic shares the adapters. That factor plausibly ranges from 0.3 to 0.9 - a 3x spread, larger than every other uncertainty combined.

The one measurement that narrows it: run all_reduce_perf from nccl-tests at the message size the job actually uses, on the real node count, and read the reported bus bandwidth. One afternoon converts a 3x range into a 1.2x range. Nothing else in this estimate is worth buying.

What the number rules in or out

If the forward and backward pass takes about 2 s, 5.6 s of flat all-reduce is not hideable and the step is communication-bound by a factor of three. That rules out the flat topology and rules in three moves, in this order: hierarchical collectives, gradient bucketing so reduction of early layers overlaps the backward pass still running on later ones, and only then reducing the number of data-parallel replicas by moving parallelism inside the node where the fabric is 18x wider.

It also rules out a cheaper fabric. Halving inter-node bandwidth to 25 GB/s doubles the cross-node term, which in the hierarchical plan is 0.70 s of a 1.3 s budget - so the saving costs over half the step. That is the trade-off the estimate exists to expose.

When this is the wrong answer

When the model fits comfortably on one node. At 7B parameters in bf16, gradients are 14 GB and the whole reduction happens over NVLink at 0.03 s; the fabric is irrelevant and this arithmetic is wasted work. The estimate earns its keep only once a replica spans nodes.