Sparse Upcycling from Dense Checkpoints

You have a trained dense model and you want the capacity of a mixture-of-experts without paying for MoE pretraining from scratch. Sparse upcycling is the conversion: for each transformer block you want to sparsify, clone the existing feed-forward network \(E\) times to form \(E\) experts, insert a freshly initialised router in front of them, and continue training so the identical clones diverge into specialists.

The original result showed upcycled T5 Base, Large and XL models, and ViT Base and Large, significantly outperforming their dense counterparts on SuperGLUE and ImageNet while using roughly 50% of the initial dense pretraining sunk cost in additional training (Komatsuzaki et al., 2023, Sparse Upcycling, ICLR, arXiv:2212.05055).

The symmetry problem

At the instant of upcycling, every expert is identical. The router's output therefore carries no information about which expert is better, because none is, and the gradient that would differentiate them is the same for all of them up to whatever routing noise exists. Specialisation has to be bootstrapped out of that degenerate state.

Two things break the symmetry in practice. The router is randomly initialised, so it assigns different tokens to different experts from step one, and each expert then receives a different token distribution and drifts accordingly. And the load-balancing auxiliary loss actively pushes tokens away from whichever expert is currently over-subscribed, which prevents the degenerate solution where the router collapses onto a single copy.

The early steps are correspondingly fragile. Too strong an auxiliary loss and routing becomes uniform, which reproduces the dense model at \(E\) times the parameter cost. Too weak and the router collapses. This is the same tension as in from-scratch MoE training, but sharper, because the experts start with nothing to distinguish them.

Scaling and granularity

Two refinements matter at production scale. The first is output scaling: after cloning, the MoE layer's output is a weighted combination of expert outputs rather than a single FFN output, so the activation statistics that the rest of the network was trained against shift unless you correct for it. The second is granularity. Rather than cloning the FFN whole into a few large experts, you can partition it into many smaller experts, which changes the combinatorics of what routing can express.

NVIDIA's study introduced a "virtual group" initialisation scheme and a weight-scaling approach to make upcycling into fine-grained MoE work, and reported that higher granularity helps accuracy and that softmax-then-topK routing beats topK-then-softmax. Their headline comparison is the honest one to quote: upcycling Nemotron-4 15B over 1T tokens reached 67.6% MMLU against 65.3% for continuing to train the same dense model on the same 1T tokens (He et al., 2024, Upcycling Large Language Models into Mixture of Experts, arXiv:2410.07524).

A 2.3-point MMLU gain for the same token budget is real and is also not enormous. Upcycling is a way of spending memory to buy accuracy at fixed training compute, and it should be evaluated on that basis.

Where it shows up

Skywork-MoE is the clearest public example of the pattern at scale: a 146B-parameter, 16-expert model initialised from the existing dense Skywork-13B checkpoints, with gating logit normalisation to improve expert diversification and adaptive auxiliary loss coefficients tuned per layer (Wei et al., 2024, Skywork-MoE, arXiv:2406.06563). The per-layer auxiliary coefficient is a telling detail: the right balancing pressure is not uniform across depth, and upcycled models in particular tend to want less of it in early layers.

When it breaks

The decision is not "upcycle or train from scratch" in the abstract; it depends on the quality of the dense checkpoint you hold and on the size of the MoE training budget you have. With a large budget, from-scratch MoE eventually wins, because the upcycled model carries the dense model's representational commitments. With a small one, upcycling wins comfortably. The crossover is empirical and moves with both quantities.

The cost that surprises teams is at inference, not training. An upcycled model has \(E\) copies of every upcycled FFN resident in memory, so a 13B dense model becomes a 146B-parameter artefact to host even though only a fraction is active per token. Reported inference slowdowns of up to around 40% for larger upcycled models, attributable to routing overhead and increased active parameters, are in the same family of surprise: sparsity reduces FLOPs, and FLOPs are not the only thing that determines latency.

Upcycling also does not fix a weak base. Every deficiency of the dense checkpoint is copied \(E\) times, and the router has no mechanism for correcting it. If the dense model is undertrained, train it further before converting it, not after.