The Critical Batch Size

Data parallelism scales by widening the batch: more replicas, more sequences per optimiser step, in principle a proportionally shorter wall-clock training time. In principle, because past a certain batch size that speedup stops being proportional. A team with twice the GPUs does not get a training run that finishes in half the time forever; at some point, doubling the batch barely reduces the number of optimiser steps needed to reach a given loss, and all those extra GPUs are computing largely redundant gradient information. The point where this transition happens is the critical batch size, and it is one of the more load-bearing, least intuitive numbers in large-scale training.

The gradient noise scale

McCandlish et al.'s empirical study of large-batch training frames this through the gradient noise scale: a single minibatch gradient is a noisy estimate of the true gradient over the full data distribution, and the ratio of that noise's variance to the true gradient's squared magnitude sets a natural batch size below which averaging more examples together genuinely sharpens the estimate, and above which it mostly averages out noise that was already small relative to the signal. Roughly:

B_crit ~ trace(gradient_covariance) / ||true_gradient||^2

Below B_crit, doubling the batch size lets you roughly halve the number of steps needed for the same progress, a close-to-linear trade of more parallel compute for less wall-clock time. Above B_crit, doubling the batch keeps buying only marginal step-count reduction; the compute cost of the wider batch stops being repaid in faster convergence.

It moves during training

B_crit is not a fixed property of a model, it is closer to a property of the current loss. Early in training, when the loss is high and gradients carry a lot of signal relative to their noise, B_crit tends to be smaller and grows as training proceeds and the easy, high-signal progress is used up; near convergence, gradients are noisier relative to their magnitude and B_crit grows further, meaning very large batches become comparatively more efficient later in training than early on. Kaplan et al.'s scaling-laws paper reports this same connection between loss level and critical batch size, tying gradient noise scale directly into the compute-optimal training framework.

The linear scaling rule, and where it stops applying

Goyal et al.'s large-minibatch SGD work established the practical corollary: when you increase the batch size by a factor k, scale the learning rate by roughly the same factor k to keep the optimisation dynamics equivalent. This rule holds well below the critical batch size. Push the batch size past B_crit and keep scaling the learning rate with it, and the result is frequently the opposite of stability: a learning rate calibrated for a batch size the gradient noise can no longer support is a well-documented trigger for loss spikes. Choosing a pretraining batch size is, in practice, choosing a point near but not far past B_crit for the compute budget at hand, then using gradient accumulation to hit that target batch size across however many GPUs are actually available.

When it falls down

• It is measured, not derived. There is no closed-form way to compute B_crit for a specific model and dataset from architecture alone; it is estimated empirically from short side experiments, and that estimate carries its own noise.
• A fixed batch size is a compromise across the whole run. Since B_crit grows as training proceeds, any single batch size chosen at the start is too conservative for late training and, in principle, too aggressive relative to what would have been efficient very early on.
• Undershooting wastes parallelism headroom. A batch size picked well below B_crit out of caution leaves data-parallel capacity on the table that could have been used with little efficiency cost.
• Overshooting wastes compute outright. A batch size well past B_crit spends GPU-hours computing gradient information that mostly duplicates what a smaller batch already captured, without a matching reduction in optimiser steps.

Further reading

• McCandlish et al., 2018, An Empirical Model of Large-Batch Training, arXiv:1812.06162 - the gradient noise scale and critical batch size framework.
• Goyal et al., 2017, Accurate, Large Minibatch SGD, arXiv:1706.02677 - the linear learning-rate scaling rule.
• Kaplan et al., 2020, Scaling Laws for Neural Language Models, arXiv:2001.08361 - connects critical batch size to loss level and compute-optimal training.