Composing Parallelism Strategies

No frontier run uses one parallelism strategy. A 400B-parameter model on 16,000 GPUs is simultaneously tensor-parallel inside a node, pipeline-parallel across a few nodes, data-parallel across the rest, and often context-parallel and expert-parallel on top. The interesting question is not what each strategy does in isolation but how they are assigned to a physical topology, because that assignment, not the strategy list, is what determines throughput.

The mesh

Think of the cluster as an \(n\)-dimensional grid of devices where each axis carries one parallelism strategy, and every device belongs to exactly one group per axis. For a mesh of \(\text{TP} \times \text{PP} \times \text{DP}\) the device count is the product, so 8-way tensor, 8-way pipeline and 256-way data parallelism occupies \(8 \times 8 \times 256 = 16{,}384\) GPUs.

The assignment rule follows one principle: the chattiest axis gets the fastest link.

Tensor parallelism inside the node is close to universal because its all-reduces are synchronous and per-layer. Push TP across nodes and the per-layer collective crosses a link an order of magnitude slower, which shows up immediately as a throughput collapse.

Choosing the shape

The usual procedure is memory-first, then efficiency:

1. Fit the model. Pick TP (and PP) large enough that parameters, gradients and optimiser state fit, or use ZeRO/FSDP sharding to spread optimiser state across the data-parallel axis instead.
2. Fit the activations. Add gradient checkpointing and, for long sequences, context parallelism.
3. Fill the pipeline. Choose micro-batch count \(m\) large enough that the bubble fraction is acceptable, which pushes toward larger global batch size.
4. Spend what remains on data parallelism, subject to critical batch size. Past that point extra data parallelism buys throughput but not convergence per token.

Step 4 is the constraint that surprises people. Scaling out is not free at the optimisation level: beyond the critical batch size, doubling the global batch roughly doubles the compute per unit of progress. Clusters get bigger; useful global batch size does not grow proportionally, which is why pipeline and tensor degrees keep rising instead.

Interaction effects

The axes are not independent. Higher TP shrinks per-device activations, which allows fewer checkpointed layers, which raises throughput, but it also raises collective volume per layer. Higher PP reduces per-device parameter memory but increases the bubble unless micro-batches increase, which raises activation memory again. FSDP's parameter all-gather overlaps well with compute at moderate sizes and becomes exposed when the model per layer is too small to hide it.

There is no closed form for the optimum. Frontier teams sweep configurations empirically on a small number of steps, and the published configurations of large runs are the practical reference points.

When it breaks

• A power-of-two obsession. The mesh must divide the device count exactly, so an awkward cluster size or a model with 60 layers and 8-way pipeline parallelism forces uneven splits and idle devices.
• Silent imbalance. Pipeline stages should hold equal compute, not equal layer counts. Embedding and output layers are heavier, so a naive equal-layer split leaves the first and last stages straggling for the whole run.
• Changing the mesh changes the numerics. Reduction order differs, so loss curves are not bit-identical across layouts. Comparing a config change against a previous run requires accepting that noise, which makes small regressions genuinely hard to detect.
• The mesh is baked into the checkpoint. Resharding a checkpoint saved under one layout for another is supported by mature frameworks and is a common source of subtle corruption in home-grown ones.