Depth and Width Model Growth

Training a 10B model from random initialisation discards every FLOP spent on the 7B model you already have. Model growth is the attempt not to: take the trained smaller network, map its parameters into a larger architecture, and continue training from there. The question is what mapping, and whether the larger model actually converges faster than it would have from scratch, or merely starts lower and then plateaus.

Function-preserving growth

The oldest line of attack is to require the expansion to be function-preserving: the grown network computes exactly the same function as the parent at initialisation, so training resumes without a loss spike. Net2Net established the pattern for width (replicate units and halve the outgoing weights, so the sum is unchanged) and depth (insert identity layers) (Chen, Goodfellow & Shlens, 2016, Net2Net, ICLR, arXiv:1511.05641).

Function preservation is appealing and also constraining, because a network that computes an identical function has identical gradients in the duplicated units and needs symmetry-breaking noise to differentiate them. The more general framing replaces the hand-designed rule with a learned linear map from small-model parameters to large-model parameters, factorised into width and depth operators with Kronecker structure. LiGO does this and reports saving up to about 50% of the compute of training from scratch, including roughly 45% FLOP savings on BERT and about 39% on GPT2-1.5B to reach the same log perplexity, while generalising StackBERT and Net2Net as special cases (Wang et al., 2023, Learning to Grow Pretrained Models for Efficient Transformer Training, ICLR, arXiv:2303.00980).

Depthwise stacking won

The systematic comparison at LLM scale sorted the field into four atomic growth operators (expand or stack, in width or depth) and evaluated them under one standardised pretraining setup. The result favoured the simplest thing in the set: a depthwise stacking operator, \(G_{\text{stack}}\), which duplicates the existing layer stack, gave the strongest acceleration, and the paper supplies scaling guidelines for when to grow and by how much (Du et al., 2024, Stacking Your Transformers, NeurIPS, arXiv:2405.15319).

Depth up-scaling is the production form of the same idea. SOLAR 10.7B took a 32-layer Mistral-architecture base, made two copies, dropped the final \(m = 8\) layers of the first copy and the first 8 of the second, and concatenated the remainder into a 48-layer model (\(2n - 2m = 64 - 16 = 48\)), then continued pretraining (Kim et al., 2024, SOLAR 10.7B, arXiv:2312.15166). Cutting the seam layers is the interesting detail: naively concatenating two full copies leaves a discontinuity where the second copy's early layers expect embedding-like inputs and receive deep representations instead, and removing the layers on either side of the join reduces that mismatch.

What gets inherited

A grown model inherits more than weights. It inherits the parent's tokeniser, its positional-encoding scheme and base frequency, its learned biases, and any contamination in the parent's corpus. It also inherits the parent's failures: if the 7B model was weak at multi-digit arithmetic because of how its tokeniser split numbers, the grown 11B model starts weak at multi-digit arithmetic for the same reason.

There is a scaling-law consequence too. The grown model's loss-versus-tokens curve is not the curve a from-scratch model of that size would follow, so compute-optimal token budgets derived for from-scratch training do not transfer directly. In practice grown models need a re-fitted schedule and an empirically chosen continued budget.

When it breaks

Growth buys a better initialisation, not a better ceiling. The clearest negative result in this literature is that many growth operators start well below a from-scratch baseline and then converge to a similar or worse loss, which is why the acceleration claim has to be stated as "reaches loss \(L\) in fewer FLOPs" rather than "reaches a lower loss".

Growth also pushes the architecture toward shapes nobody chose. A 48-layer model produced by stacking has whatever depth-to-width aspect ratio falls out of the arithmetic, which may be far from what a clean design would pick at that parameter count, with real consequences for inference latency (depth is serial) and for pipeline-parallel partitioning.

Finally, the seam matters. Reports of depth growth consistently find that where you cut and join determines whether the continued run recovers quickly or spends a large fraction of its budget repairing the discontinuity.