InfiniBand and Inter-Node Networking

A single H100 GPU delivers roughly 3.35 TFLOPS of FP64 throughput but connects to the rest of the cluster through a wire rated at perhaps 200 Gbps. When you are synchronising gradients across thousands of such cards, the arithmetic units often idle, waiting for bytes to arrive. Inter-node networking is not a footnote to the accelerator story; for many workloads it is the binding constraint.

What InfiniBand actually is

InfiniBand (IB) is a switched-fabric interconnect standard defined by the InfiniBand Trade Association. Unlike Ethernet, which was designed for bursty packet traffic with loose latency guarantees, IB was engineered from the start for consistent sub-microsecond latency and deterministic throughput.

Each generation roughly doubles the per-4x-link signalling rate:

The adapter latency drops from about 5 microseconds for early SDR hardware to under 0.6 microseconds for HDR. That sub-microsecond latency is not marketing; it matters for small-message collectives where synchronisation round trips dominate.

The defining feature is RDMA (Remote Direct Memory Access). With RDMA, a NIC can read from or write to a remote node's pinned memory without involving the remote CPU. The initiating CPU posts a work request; the hardware completes the transfer; a completion queue entry signals done. CPU overhead per transfer collapses from O(bytes) to O(1) for large payloads. This is why IB and its Ethernet cousin RoCE (RDMA over Converged Ethernet) dominate AI training clusters: gradient tensors can be moved peer-to-peer at wire speed without paying kernel overhead on every packet.

Fat-tree topology and why it matters

The physical switching architecture determines whether bandwidth compounds or contends.

A fat-tree builds a multi-root tree where each layer of switches has more upward-facing ports than the layer below it, guaranteeing full bisection bandwidth: any communication pattern between equal-sized halves of the cluster is served at full line rate simultaneously. In a two-layer fat-tree, leaf switches connect directly to servers; spine switches connect leaf switches to one another. The DeepSeek Fire-Flyer cluster, for example, uses a two-layer fat-tree with Mellanox QM8700 switches (40 ports at 200 Gbps each), and found this was significantly cheaper than the three-layer topology typical of larger DGX-based deployments while still providing adequate bandwidth for their workloads.

The alternative, cheaper topology is multi-rail with oversubscription: each server has multiple NICs attached to different switches, but total uplink bandwidth is smaller than total downlink bandwidth. This is fine if you can co-schedule communicating nodes on the same leaf pod. When they span pods, bandwidth is reduced and latency spikes.

A practical design choice: multi-rail with eight 200 Gbps NICs per server (as used in the MegaScale deployment at ByteDance) provides both redundancy and aggregate bandwidth without requiring a three-layer fabric, provided ECMP (Equal-Cost Multi-Path) hashing is tuned to avoid hot-spot collisions.

Collective communications and NCCL

Training rarely requires raw point-to-point transfers. The primitives that drive distributed learning are collectives: operations across a process group where each participant both contributes data and receives a combined result.

The critical ones are:

• AllReduce: every rank contributes a tensor; every rank receives the element-wise sum (or mean). Used for data-parallel gradient synchronisation.
• AllGather: every rank contributes a shard; every rank receives the concatenation of all shards. Dominant in ZeRO-3 and tensor parallelism.
• ReduceScatter: the logical inverse of AllGather; rank i receives the sum of slice i from all participants. Paired with AllGather, it forms the two-phase AllReduce used in ring-based algorithms.

NVIDIA's NCCL (Collective Communications Library) sits between the application and the hardware. It detects the physical topology at initialisation, selects ring, tree, or direct algorithms based on message size and node count, and dispatches transfers over NVLink (intra-node) or IB / RoCE (inter-node) without application changes. NCCL's auto-topology detection covers PCIe, NVLink, NVSwitch, InfiniBand, and RoCE.

Ring-AllReduce illustrates the bandwidth-optimal strategy. Arrange N nodes in a ring. In a scatter-reduce phase, each node passes 1/N of its data to its right neighbour, accumulating partial sums; after N-1 steps every node holds the correct reduced value for one 1/N shard. In an allgather phase, the shards circulate again so every node ends up with the full result. Total traffic per node: 2(N-1)/N times the tensor size, converging to 2x for large N regardless of cluster size. This is why ring-AllReduce scales well: adding more nodes does not increase per-node traffic.

The DeepSeek team's custom HFReduce implementation achieved 6.3-8.1 GB/s effective inter-node bandwidth for a 186 MiB AllReduce at scales from 16 to 1,440 GPUs, compared to 1.6-4.8 GB/s from NCCL under similar conditions - a 2-4x improvement gained by tuning NVLink-based intra-node aggregation before crossing the IB fabric.

Compute-communication overlap

Waiting for gradients to finish before starting the next forward pass wastes the GPU. The standard mitigation is pipeline overlap: begin AllReduce on the gradients of early layers while the backward pass is still computing gradients for later layers. Frameworks partition the model's parameters into buckets; once a bucket's gradients are complete, its AllReduce is launched asynchronously.

The condition for full overlap is that the communication time for each bucket is shorter than the computation time for the subsequent bucket:

T_comm(bucket_k) < T_compute(backward, layers > k)

When this holds, network latency disappears from the critical path. When it does not - because the model is shallow, the batch is small, or the network is slow relative to the GPU - communication becomes a serial bottleneck. This is the situation that motivates pushing from EDR to HDR to NDR InfiniBand: not raw throughput per se, but the ability to keep overlap conditions satisfied as GPU FLOPS outpace link speeds.

When it falls down

Hot-spot contention with ECMP: Equal-Cost Multi-Path hashing distributes flows across uplinks. If hashing is purely 5-tuple based and many flows share source/destination IP pairs (common in gradient sync), they land on the same uplink and contention degrades effective bandwidth. Fixes include flow-aware routing, ECMP with 7-tuple hashing, or traffic matrix scheduling at the job level.

Link flapping and silent errors: IB links occasionally degrade or flap without failing completely. Dropped completions manifest as hanging AllReduces because a single slow rank blocks the collective. MegaScale reports that NCCL's retransmit timer and retry count are critical tuning knobs; setting them too aggressively causes false positives, too conservatively masks real failures.

RoCE congestion cascades: RoCE traffic is sensitive to switch buffer exhaustion. A brief burst fills a port buffer, triggers pause frames (PFC), which propagate back to senders, causing head-of-line blocking that can freeze an entire fat-tree pod. Combining Swift and DCQCN congestion control, as MegaScale does, reduces this, but it requires careful per-hop configuration.

Initialisation at scale: NCCL rendezvous and route setup for 10,000+ GPUs can take over 15 minutes with naive implementations. MegaScale reduced this to under 30 seconds through parallel bootstrap and deferred route computation, illustrating that protocol overhead, not just steady-state throughput, is a real operational constraint.

Topology mismatch for tensor parallelism: Tensor-parallel AllReduces between GPUs on different nodes generate many small, latency-sensitive messages rather than large bandwidth-bound ones. IB latency under 1 microsecond is adequate; crossing a slow Ethernet segment is not. Placement policies must ensure that tensor-parallel groups stay on the same high-speed fabric zone.

Further reading

• Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning - DeepSeek's detailed account of building a large IB cluster, including HFReduce design and fat-tree topology choices.
• MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs - ByteDance's production engineering at scale, covering multi-rail networking, ECMP congestion, and link flapping.
• NCCL Documentation - Reference for collective operation semantics, supported transports, and environment-variable tuning knobs.
• InfiniBand - Wikipedia - Concise generation-by-generation bandwidth and latency specs with primary standards references.