Mixture-of-Experts Inference

MoE models look like a serving dream: DeepSeek-V3 activates 37B of its 671B parameters per token, so it should serve like a 37B dense model. In training, broadly, it does. In inference it absolutely does not. You still have to hold all 671B parameters in memory because any token might route to any expert, expert routing creates load imbalance, small batches catastrophically under-utilise the GPUs, and expert-parallel sharding adds an all-to-all communication step that dense models never pay. The throughput win is real, but the engineering bill is significant.

Why serving MoE is harder than dense

A dense model's inference cost is weights_read + kv_read per token. Predictable, contiguous, friendly to bandwidth optimisation. An MoE model adds three new problems:

1. You still hold all experts in HBM. Active parameters dictate FLOPs, but total parameters dictate memory. Mixtral 8x7B is 47B parameters; you cannot serve it on a 24 GB GPU even though only ~13B activate per token.
2. Routing is data-dependent. The router decides per token which experts to fire. You discover the GEMM shapes at runtime, which fights every static-shape optimisation a compiler wants to do.
3. Load is rarely balanced. Even with auxiliary balance losses, popular experts get 2-3x the load of cold ones during inference. The slowest expert sets the step time.

Top-k routing and the memory bill

Most production MoE models use top-2 routing: each token activates 2 of N experts. Memory budget for inference:

You read only the active experts' weights per token, so decode bandwidth is bounded by active parameters - that is the whole point. But you cannot evict cold experts cheaply because the next token may route to them. Holding the full model resident is non-negotiable for any non-trivial throughput.

The small-batch problem

This is the one that surprises people. Suppose batch size is 1 and you generate one token. The router picks 2 of 256 experts. Those 2 experts each do a single matmul on a single token's hidden state - call it a 1-by-d-by-h GEMM. Tensor cores want big M dimensions; 1 is the worst possible shape. Meanwhile the other 254 experts sit idle, taking up HBM and contributing nothing.

You loaded the full model. You got the FLOPs of a 1-token, 2-expert decode. Effective utilisation is in the low single digits.

The fix is batch size. With batch 256, each expert sees on average ~2 tokens, and with batch 2048 each sees ~16 - now the matmul shapes are healthy and the GPU stays busy. MoE serving fundamentally requires high batch sizes to be efficient. This is the opposite of what many low-traffic deployments want.

Expert parallelism

You cannot fit DeepSeek-V3 on one GPU at any precision. You shard. The natural sharding is expert-parallel: put expert i on GPU i % world_size. Then for each token, the layer:

1. Computes the router scores on every GPU.
2. All-to-alls the token's hidden state to the GPU holding its assigned expert.
3. Runs the expert matmul.
4. All-to-alls the result back.

That all-to-all is the dominant cost of MoE serving at scale. It scales with hidden dim and batch size, and it stalls every GPU until the slowest one finishes. NVLink and NVSwitch make it tolerable inside a node; cross-node all-to-all over InfiniBand is where MoE serving gets painful.

The usual production layout: expert-parallel within a node (8 H100s, NVLink), data-parallel across nodes. DeepSeek-V3's deployment goes further with prefill-decode disaggregation (separate clusters for the two phases) so that the all-to-all patterns can be tuned independently.

How DeepSeek and Mistral made it work

The two production playbooks look different.

Mistral / Mixtral. Coarse-grained: 8 experts of full FFN size, top-2 routing. The expert kernels look enough like dense FFNs that vLLM, TensorRT-LLM, and SGLang can serve them with modest changes. The cost is that experts duplicate a lot of knowledge - 8 experts is not much specialisation - so per-active-parameter quality is lower than DeepSeek's regime.

DeepSeek (V2/V3/MoE). Fine-grained: many small experts (256 in V3) plus shared experts that every token uses. Their argument (DeepSeekMoE paper): smaller experts specialise better, shared experts capture common patterns, and the combination delivers more capability per active parameter than coarse routing. The cost is harder kernels (more, smaller matmuls), heavier all-to-all traffic (more routing destinations), and a co-designed training stack. DeepSeek shipped FP8 weights, fused expert kernels, MLA for the KV cache (cuts cache size 4-8x), and prefill-decode disaggregation. None of these are optional; they are what makes the architecture servable.

Llama-4's Maverick and Behemoth use 128 experts plus 1 shared, broadly following the DeepSeek direction.

Expert offloading to CPU

For low-batch local serving, you can hold cold experts in CPU RAM and stream them in over PCIe when called. Frameworks like Mixtral-Offloading and DeepSpeed-MoE-Inference do this. The numbers are sobering:

• PCIe Gen5 x16: ~64 GB/s. Loading one Mixtral expert (~5.5 GB) takes ~85 ms. That is your time-per-token floor, against ~30 ms on-GPU.
• Mitigations: LRU-cache recent experts on the GPU, prefetch based on router predictions, quantise the offloaded copy to INT4.

Useful for "run a 47B MoE on a 24 GB consumer GPU" demos. Not a production strategy.

When MoE inference makes sense

• High-throughput serving with concurrent users. Batch sizes above 256 make the math work. Frontier-lab serving and bulk-batch workloads benefit most.
• You have multi-GPU NVLink nodes. The all-to-all has to land somewhere fast.
• Your quality target requires the parameter count. If a dense 70B is good enough, serve a dense 70B; you avoid a whole class of operational pain.

When it falls down

• Low QPS, latency-critical workloads. Single-stream MoE serving is dominated by routing overhead and under-batched GEMMs. A dense model of equivalent active size will beat it.
• Cross-node expert parallelism on slow interconnects. Without InfiniBand or NVLink-class fabric, the all-to-all dominates and the GPUs starve.
• Long-context dominated workloads. MoE saves FFN compute, not attention compute. At 128k context, attention dominates and the MoE win shrinks.

Further reading

• Mixtral of Experts - the production-ready coarse-grained MoE that put the architecture on every serving stack.
• DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models - the fine-grained + shared-expert design that V2 and V3 build on.
• DeepSeek-V3 Technical Report - the full serving stack: FP8, MLA, prefill-decode disaggregation, expert parallelism.
• vllm-project/vllm GitHub - reference implementation of expert-parallel MoE serving, including fused MoE kernels.