Topology-Aware Scheduling

An H100 exposes 900 GB/s of NVLink bandwidth, a figure that counts both directions, so roughly 450 GB/s each way (NVIDIA, H100 GPU). A 400 Gb/s InfiniBand port carries 50 GB/s. The same collective can therefore run nine times slower depending on which GPUs the scheduler picked, and nothing in a "64 GPUs" resource request says so.

Gang scheduling and fragmentation establishes that a job must get all its GPUs at once and in the right shape. This concept goes a level down: how to score a placement, how fabrics are wired to make good placements possible, and how to map parallelism onto the hierarchy.

The cost model

For ring all-reduce of an \(n\)-byte buffer over \(p\) ranks, with per-hop latency \(\alpha\) and bottleneck bandwidth \(B\) in bytes per second, the time is approximately

The bandwidth term dominates for large gradients, and \(B\) is the slowest link anywhere in the ring. All-reducing a 10 GB bucket across 8 GPUs costs \(2 \times \tfrac{7}{8} \times 10 / 450 \approx 0.039\) s inside an NVLink domain and \(17.5 / 50 = 0.35\) s if the ring crosses a 400 Gb/s fabric link. These are idealised: tree algorithms, congestion and in-network reduction change the constants, but not the conclusion that one bad hop sets the pace.

How fabrics are wired

In a rail-optimised fabric, GPU \(k\) of every server connects through its own NIC to leaf switch \(k\), the "rail". NVIDIA's DGX SuperPOD reference groups 32 nodes into a scalable unit where traffic on a rail "is always one hop away from the other 31 nodes", while traffic between rails traverses the spine (NVIDIA, DGX SuperPOD Reference Architecture). Collective libraries exploit this by keeping each ring or tree on one rail and using NVLink inside the server to switch rails.

Wang and colleagues argue LLM training traffic is sparse enough that the spine is largely unnecessary: a rail-only design, forwarding cross-rail traffic through NVLink, cut network cost by 38-77 percent and network power by 37-75 percent with training performance maintained, and added only 8.2-11.2 percent completion-time overhead for mixture-of-experts all-to-all (Wang et al., 2023, Rail-only, arXiv:2307.12169). Whether to give up any-to-any connectivity is a live disagreement. Operators running many tenants and unpredictable job mixes value a full-bisection fabric's flexibility; operators running a few giant training jobs see the spine as cost that the traffic does not use.

Scale-up domains are growing too. A GB200 NVL72 rack puts 72 GPUs in one NVLink domain at 1.8 TB/s per GPU (NVIDIA, GB200 NVL72), which moves the fast-placement unit from an 8-GPU server to a rack.

Mapping parallelism onto the hierarchy

Different parallelism dimensions tolerate different links. Tensor parallelism all-reduces activations in every layer and belongs inside an NVLink domain. Expert parallelism's all-to-all is similarly chatty and benefits from large scale-up domains. Pipeline parallelism sends point-to-point activations between stages and tolerates a same-leaf hop. Data parallelism communicates once per step, overlaps communication with backward computation, and can span the spine.

A scheduler scores a candidate placement \(\pi\) by expected communication cost,

where \(v_{ij}\) is traffic volume between ranks \(i\) and \(j\) implied by the parallelism plan, \(\ell\) is the lowest common level of the topology (same GPU domain, same leaf, same spine block), and \(c\) is that level's cost. Placing the job means minimising \(C\) subject to availability, which in practice is a greedy search: fill whole NVLink domains, then whole leaves.

When it breaks

Good placement costs waiting. Microsoft's analysis of its Philly cluster found that gang scheduling and locality constraints drive queueing delay for DNN jobs, while jobs spread across servers train more slowly (Jeon et al., 2019, Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads, USENIX ATC). The right policy compares waiting time against the slowdown multiplied by runtime; a two-hour job should not wait six hours for a 10 percent faster placement.

Topology the scheduler does not know is topology it ignores. A failed cable, a degraded optic or adaptive routing around congestion changes effective bandwidth without changing the declared graph. Placement needs live link health, not just the wiring diagram.

Rank order matters as much as node choice. Well-chosen nodes with a launcher that assigns ranks in hostname order can put tensor-parallel peers on different servers. Placement must extend to the rank-to-GPU mapping.

Fragmentation feeds back. Strict topology packing leaves partial domains and partial leaves stranded, so the cluster's topology-feasible capacity for the next large job is smaller than its free GPU count.