Pipeline Bubbles and Schedules

Split a model into \(p\) sequential stages on \(p\) devices and the first problem is obvious: stage 2 cannot start until stage 1 produces something, and stage 1 cannot start the backward pass until every later stage has finished forward and returned a gradient. Run one batch through and each device is busy for roughly \(1/p\) of the wall clock. Pipeline parallelism only works if that idle time, the bubble, can be filled.

The bubble fraction

Split the batch into \(m\) micro-batches and feed them in sequence. Devices fill up as micro-batches arrive and drain at the end, so with \(p\) stages the idle fraction is

With \(p = 8\) and \(m = 8\), that is 47 percent of device time wasted. With \(m = 64\) it falls to about 10 percent. This single expression governs the design: more micro-batches means less bubble, which is why pipeline parallelism pushes global batch size upward, and why it interacts directly with critical batch size.

The cost of raising \(m\) is activation memory. Every micro-batch in flight holds its forward activations until its backward pass runs.

From GPipe to 1F1B

GPipe's schedule runs all \(m\) forward passes, then all \(m\) backward passes. Simple, and it holds up to \(m\) micro-batches' worth of activations on the first stage at peak.

1F1B (one forward, one backward) interleaves them: once the pipeline is full, each device alternates a forward micro-batch with a backward one. The bubble fraction is identical, but peak activation memory drops to roughly \(p\) micro-batches instead of \(m\), because a micro-batch's activations are freed as soon as its backward pass completes. Same throughput, far less memory, which is why 1F1B is the default in Megatron-LM and DeepSpeed.

Interleaved 1F1B goes further by giving each device several non-contiguous chunks of layers (device 0 holds layers 1-2 and 17-18, and so on). With \(v\) chunks per device the bubble shrinks by roughly a factor of \(v\):

The cost is \(v\) times more pipeline communication, since activations cross device boundaries more often.

Zero-bubble schedules exploit the fact that the backward pass has two separable halves: the gradient with respect to the input, which the previous stage needs immediately, and the gradient with respect to the weights, which is needed only before the optimiser step. Deferring the weight gradients fills the remaining bubble with useful work, in principle reducing it to near zero at the cost of a more complex scheduler and higher memory pressure.

Where the time actually goes

A pipeline stall is not always a bubble. Three separate effects look identical on a throughput chart:

• The bubble, which is structural and predicted by the formula above.
• Stage imbalance, where one stage takes longer per micro-batch and every other device waits for it every step. Balancing by layer count rather than by compute is the usual cause, since embedding and loss layers are heavier than transformer blocks.
• Exposed communication, where the activation transfer between stages does not overlap with compute.

Only the first is fixed by more micro-batches. Diagnosing which one you have requires a profile, not a formula.

When it breaks

• Memory forces small \(m\). If activations at \(m = 64\) do not fit, the bubble you calculated is not the bubble you get. Gradient checkpointing buys micro-batches at the cost of recomputation.
• The last stage carries the loss. Loss computation, and often the unembedding matrix, sit on the final stage and make it the straggler. Splitting the embedding or moving the loss is a standard fix and a standard oversight.
• Interleaving multiplies inter-node traffic. On a slow fabric, the communication added by \(v\) chunks can exceed the bubble it removed.
• Recovery is expensive. A failure on any stage stalls the whole pipeline, and pipeline state is harder to restore than replicated data-parallel state. See fault tolerance at scale.