Depth Versus Width Tradeoffs
Why two models with identical parameter counts behave differently depending on how the parameters are arranged, what depth buys that width cannot, and how the hardware votes for width.
Two seven-billion-parameter transformers: one with 32 layers of width 4,096, one with 64 layers of width 2,900. Same parameter count, roughly the same training FLOPs, materially different models. Loss curves separate, capability profiles separate, and inference cost separates in a way that has nothing to do with the parameter count. The aspect ratio of a model is a design decision, and it is usually made by copying whatever the last successful model did.
What depth provides
Depth is sequential computation. Each layer reads the residual stream, transforms it, and writes back, so a model with \(L\) layers can perform at most \(L\) dependent steps of computation on a single forward pass. A task requiring a chain of five inferences needs enough depth to host the chain; no amount of width supplies it, because width parallelises within a step rather than adding steps.
This shows up concretely. Multi-step arithmetic, multi-hop retrieval within context, and compositional reasoning all correlate with depth more strongly than with total parameters. The circuits interpretability work makes the mechanism visible: induction heads, for instance, require at least two attention layers in sequence, because one layer must write the information the next layer reads.
What width provides
Width is representational capacity per step: how much information the residual stream carries, how many features can be represented without interference, and how many attention heads can specialise. Wider models store more factual knowledge for the same depth, and knowledge-heavy benchmarks track width and total parameters more than depth.
Width also parallelises. A wider matmul is a larger, more efficient GEMM; a deeper stack is more sequential kernel launches, each with fixed overhead and each a synchronisation point. This matters enormously for inference latency, where the per-layer overhead is paid once per token per layer and cannot be batched away.
Where the optimum sits
Empirically, transformer language models cluster around an aspect ratio of \(d_{\text{model}} / L\) between roughly 100 and 200, and the range is stable across two orders of magnitude of scale. Kaplan et al.'s scaling work found loss to be remarkably insensitive to shape within that band, which is a useful licence: if the loss barely moves, choose shape on other grounds.
Those other grounds usually favour width. Tensor parallelism shards along the width dimension with one all-reduce per layer, so a wider, shallower model needs fewer communication rounds for the same parameter count. Pipeline parallelism shards along depth and introduces bubbles proportional to stage count. Inference latency is roughly linear in layer count at batch size one. All three push toward wide and shallow, and this is why production models are wider than the loss-optimal shape would suggest.
When it breaks
Very deep and narrow trains badly. Below an aspect ratio of about 50, models become unstable: gradient signal degrades through the stack, and the number of residual writes into a narrow stream causes interference between features. Deep-narrow configurations need extra normalisation care and often more warmup.
Very wide and shallow saturates. Above roughly 300, adding width stops helping because the model has run out of sequential steps to use the extra capacity. The loss curve flattens in a way that looks like a data limit and is a shape limit.
Reasoning-heavy post-training changes the answer. A model that will be trained to produce long chains of thought can substitute serial tokens for serial layers, doing multi-step computation across the sequence rather than across the stack. This weakens the argument for depth, and it is a recent enough shift that most published shape guidance predates it.
MoE breaks the parameter-count intuition entirely. A sparse model's depth is its layer count and its width is the active expert width, so total parameter count says nothing about aspect ratio. Comparing a dense model to an MoE by parameters is the wrong axis; active parameters and layer count are the right ones.
12 flashcards for this concept
Click a card to reveal the answer.