One Training Run, Many Models: How Elastic Architectures Replaced the Model Family

The Llama 3 herd consumed 39.3 million H100 hours (Meta, 2024, The Llama 3 Herd of Models, arXiv:2407.21783). Three models, three pretraining runs, three data pipelines, three sets of evaluations, three sets of bugs. Nothing in the architecture required that. The 8B and the 70B are the same design at different widths and depths, aimed at different memory budgets, and they were trained as if they were unrelated projects.

By late 2025 the alternative had a number attached. Nemotron Elastic embedded a 9B and a 6B model inside a 12B parent using 110B training tokens, and reported roughly 360x lower cost than training that family from scratch, with each nested model matching or beating the state of the art at its size (Nemotron Elastic, 2025, arXiv:2511.16664). The children are not separate checkpoints. They are regions of the parent's weights, extractable zero-shot at deployment.

Why this matters: The question "which size should we train?" is being replaced by "which sizes should we be able to extract?" That changes how pretraining budgets are allocated, how serving stacks are built, and which evaluations you owe. It also introduces a failure mode that catches teams repeatedly: elastic models save compute and latency, and they do not save the memory people assume they save.

TL;DR

• Elastic architectures train one parent whose sub-networks are nested inside its weights by construction, so smaller models are sliced out with no fine-tuning rather than trained separately.
• MatFormer nests the FFN at four granularities per layer. With 32 layers that is \(4^{32}\) extractable configurations, not four, because granularity is chosen per layer.
• Extracted sub-models are competitive with independently trained models of the same size, and at 850M scale MatFormer's sub-models beat their independently trained baselines, because small granularities regularise the large one.
• The saving is real and large: Flextron converts a trained LLM into an elastic one for 7.63% of original pretraining tokens; Nemotron Elastic reports about 360x versus from-scratch families and about 7x versus iterative compression.
• Memory does not shrink. A 6B child of a 12B parent still needs the parent resident, because the child's weights are a slice of it. You buy FLOPs and latency, not footprint.
• Slicing the FFN leaves the KV cache untouched. In the worked example below, a 32k-token cache is 4.3 GB against 1.4 GB of int4 weights, so the elastic axis is not the one that matters for long context.
• Gemma 3n shipped the consumer version: MatFormer plus per-layer embedding offload plus cross-layer KV sharing, giving a roughly 5B-parameter model that runs in about 2 GB of accelerator memory.
• The honest cost is evaluation. Hundreds of extractable models means hundreds of uncertified ones, and teams respond by certifying a handful, which quietly restores the fixed ladder of sizes the architecture was meant to replace.

At a Glance

flowchart LR
    D["Training data"] --> P["One parent training run"]
    P --> W["Nested weights<br/>per-layer granularities"]
    W --> S["Budget selector<br/>Mix-n-Match search"]
    S --> A["Phone build<br/>2.04B active"]
    S --> B["Laptop build<br/>2.81B active"]
    S --> C["Cloud build<br/>5.22B active"]
    W -.->|"all weights stay resident"| M["Parent footprint<br/>unchanged"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff

    class D,P blue
    class W,S purple
    class A,B,C teal
    class M amber

Before Elasticity: Three Ways to Get a Small Model

For most of the deep learning era the small model was a separate artefact. You trained it from scratch, or you compressed a big one, and either way you ended up with its own file, its own config, its own regression suite. Training from scratch is the honest option and the expensive one, and it carries a subtle inefficiency: compute-optimal scaling laws tell you how to spend a budget on one model, and say nothing useful when four models share a budget and the small ones will serve a thousand times more requests than the large one.

Compression came next and worked better than anyone expected. Structured pruning followed by distillation from the uncompressed parent is now a standard recipe: Minitron derives 8B and 4B models from a pretrained 15B using up to 40x fewer training tokens per model than training from scratch, saving 1.8x on the family's total compute and landing up to 16% better on MMLU than the from-scratch 4B (Muralidharan et al., 2024, Compact Language Models via Pruning and Knowledge Distillation, arXiv:2407.14679). That is a large win, and it still produces N independent checkpoints.

The third line is older and came from computer vision. Slimmable networks trained a single CNN that could run at several widths at runtime, using switchable batch normalisation so each width had correct normalisation statistics, and matched or beat individually trained MobileNet and ResNet variants at each width (Yu et al., 2018, Slimmable Neural Networks, ICLR 2019, arXiv:1812.08928). Once-for-All generalised it to depth, width, kernel size and resolution together, and introduced progressive shrinking to stop the sub-networks fighting each other during training (Cai et al., 2019, Once-for-All, ICLR 2020, arXiv:1908.09791). Both were pitched as neural architecture search without the search cost, which undersold them. The real result was that one set of weights can serve many budgets.

Matryoshka Representation Learning carried the idea into embeddings, training representations whose prefixes are themselves good representations, and reporting up to 14x smaller embeddings at equal ImageNet-1K accuracy (Kusupati et al., 2022, Matryoshka Representation Learning, NeurIPS 2022, arXiv:2205.13147). That is the direct ancestor of what happened to transformers next, and the reason this area's vocabulary is full of nesting dolls (see Matryoshka representation learning).

timeline
    title From Slimmable CNNs to Elastic Reasoning Models
    2018 : Slimmable Networks run one CNN at several widths with switchable BatchNorm
         : Universal Transformers loop one block instead of stacking distinct layers
    2019 : Once-for-All adds depth, kernel and resolution elasticity with progressive shrinking
    2022 : Matryoshka Representation Learning nests embeddings so prefixes stay useful
    2023 : MatFormer nests the FFN and introduces Mix-n-Match across layers
    2024 : Flextron converts a trained LLM into an elastic one for 7.63 percent of pretraining tokens
    2025 : Gemma 3n ships MatFormer on phones
         : Nemotron Elastic nests 6B and 9B inside a 12B parent
    2026 : Star Elastic adds nested submodels to hybrid reasoning models after pretraining

[IMAGE: Two-panel comparison. Left panel, "Family as N projects": four separate boxes labelled 2B, 4B, 9B, 12B, each with its own data pipeline, training run, checkpoint file and eval suite, with a total cost bar. Right panel, "Family as one artefact": one 12B box with three nested regions highlighted inside it, one training run, one checkpoint file, and three eval suites still required. Caption: "Elasticity collapses the training column and leaves the evaluation column alone."]

How Nesting Actually Works

The mechanism is easier than the name suggests, and the difficulty lives entirely in the training procedure rather than the architecture.

The nested FFN

Take a transformer block whose feed-forward network has hidden dimension \(m\). MatFormer defines a set of granularities \(G = \{m/8,\ m/4,\ m/2,\ m\}\) and treats each as a prefix of the full hidden dimension: the \(m/8\) sub-block uses the first \(m/8\) hidden units, the \(m/4\) sub-block contains the \(m/8\) one, and so on. Formally, with \(T_i\) denoting the FFN restricted to its first \(g_i\) hidden units,

so a single pair of matrices \(W_1, W_2\) serves every granularity. Nothing is duplicated, and nothing is masked at runtime: you slice the weight matrices and run a smaller matmul (Devvrit, Kudugunta et al., 2023, MatFormer: Nested Transformer for Elastic Inference, NeurIPS 2024, arXiv:2310.07707).

The prefix constraint is what makes it work. If the sub-networks were arbitrary subsets, there would be no single weight layout that serves all of them, and you would be back to storing separate models. Prefixes give you nesting for free in memory layout: a smaller model is a contiguous read.

Training all granularities at once

The loss is the part that costs something. At each step the model is trained on several granularities jointly, so the same parameters receive gradient from the \(m/8\) path and the \(m\) path simultaneously. This is where MatFormer, Once-for-All and slimmable networks all meet the same problem: the small configurations and the large one do not want the same weights.

Two things follow, and only one of them is bad. The obvious cost is interference at the top end, which is why Once-for-All introduced progressive shrinking, training the largest configuration first and admitting smaller ones gradually. The less obvious effect is that the small granularities act as a regulariser. At 850M parameters, MatFormer's extracted sub-models outperformed independently trained models of the same size on validation loss and one-shot downstream evaluations. At 2.6B, extracted models from 1.5B to 2.6B were comparable rather than better. The regularisation benefit is real and it is largest where the model is most over-parameterised for its data.

Mix'n'Match: why four granularities give more than four models

The step that turns a ladder into a continuum is choosing granularity per layer instead of globally. A 32-layer model with four trained granularities has \(4^{32}\) possible configurations, which is about \(1.8 \times 10^{19}\). The overwhelming majority were never explicitly optimised, and they work anyway, because every per-layer sub-block was trained to be a functioning FFN in some configuration.

Mix'n'Match is the heuristic that searches this space under a parameter or latency budget, and it costs nothing at deployment. The practical consequence is that you stop shipping a ladder of sizes and start shipping a function from budget to model. A phone with 3 GB free picks one configuration; the same phone under memory pressure picks another, from the same weights.

flowchart TB
    B["Budget: 2.81B params"] --> SEL["Per-layer selector"]
    ATT["Attention<br/>fixed, never sliced"] --> SEL
    subgraph FFN["Nested FFN hidden units, one weight matrix"]
        G1["first m/8"]
        G2["first m/4, contains m/8"]
        G3["first m/2, contains m/4"]
        G4["full m, contains m/2"]
    end
    SEL --> G1
    SEL --> G2
    SEL --> G3
    SEL --> G4
    G3 --> OUT["Chosen config<br/>11 layers at m/2, 21 at m/4"]
    G2 --> OUT

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class B,SEL blue
    class G1,G2,G3,G4 purple
    class OUT teal
    class ATT slate

Adding a router, and the reasoning-model version

MatFormer leaves selection to a deployment-time heuristic. Flextron goes further and trains an input-adaptive router that sends individual tokens through different sub-networks, converting an existing trained LLM into an elastic one using 7.63% of original pretraining tokens (Cai et al., 2024, Flextron: Many-in-One Flexible Large Language Model, ICML 2024, arXiv:2406.10260). At that point the line between elasticity and mixture-of-experts gets thin: both route tokens to a subset of parameters. The distinction that survives is which axis is nested. MoE experts are parallel and disjoint; elastic granularities are nested and ordered.

Nemotron Elastic took the router approach to reasoning models, with an end-to-end trained router, a two-stage curriculum, group-aware SSM elastification so that Mamba's structural constraints survive slicing, and distillation that optimises several budgets simultaneously. Star Elastic, in 2026, made it a post-training procedure on hybrid Mamba-Transformer-MoE models and used the elasticity within a single request, running the thinking phase at one budget and the answering phase at another (Star Elastic, 2026, arXiv:2605.07182). That is the first version of this idea that is not really about deployment targets at all. It is about spending less compute on the easy parts of one answer.

[IMAGE: Line chart, x-axis extracted model size from 1.5B to 2.6B, y-axis validation loss. Three series: independently trained baselines (points only), MatFormer's four explicitly trained granularities (large markers), and Mix'n'Match configurations (many small markers filling the gaps between granularities, forming a near-continuous curve). Caption: "Four trained granularities produce a continuum, not a ladder."]

Seeing It in Motion

Two views of the same system: what a serving stack does with an elastic checkpoint, and what changes when the budget changes mid-flight.

sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant S as Selector
    participant E as Elastic engine
    C->>G: request with latency SLO
    G->>S: budget hint plus device class
    S->>S: look up config for budget
    S->>E: activate config (layer granularity vector)
    Note over E: same resident weights,<br/>smaller matmul slices
    E-->>G: tokens
    G-->>C: response
    C->>G: second request, tighter SLO
    G->>S: new budget
    S->>E: activate smaller config
    Note over E: no weight load,<br/>no process restart
    E-->>C: response

The thing worth noticing in that diagram is the absent step. There is no model load between the two requests. That is the entire operational argument for elasticity over compression: a compressed family requires you to decide, at deployment time, which checkpoint a process serves, and switching means loading gigabytes. An elastic checkpoint changes a slice index.

stateDiagram-v2
    [*] --> FullBudget
    FullBudget --> ReducedBudget: memory pressure or SLO breach
    ReducedBudget --> MinimalBudget: battery saver or thermal throttle
    MinimalBudget --> ReducedBudget: pressure clears
    ReducedBudget --> FullBudget: plugged in, idle accelerator
    FullBudget --> [*]: session ends
    note right of MinimalBudget
        Weights identical in all states.
        Only the granularity vector changes.
    end note

[IMAGE: Side-by-side timeline of two serving stacks handling the same budget change at t=0. Top track, compressed family: request routed to a different process, checkpoint load of 5.6 GB, cold start, first token at t+8s. Bottom track, elastic checkpoint: granularity vector swapped in place, first token at t+40ms. Caption: "The operational difference is a checkpoint load, and it is three orders of magnitude."]

Watch It Run

By the Numbers

Sources: Llama 3 (Meta, 2024), Minitron (Muralidharan et al., 2024), Flextron (Cai et al., 2024), Nemotron Elastic (2025), MatFormer (Devvrit, Kudugunta et al., 2023), MRL (Kusupati et al., 2022). Cost ratios are each paper's own accounting against its own baseline and are not directly comparable across rows: "360x" and "1.8x" measure different things (one family against from-scratch training of that family, one against a differently sized baseline). Treat the column as evidence of direction, not as a league table.

[IMAGE: Horizontal bar chart of training tokens required to produce a model family, log scale. Bars for from-scratch family, prune-and-distil, Flextron conversion, Nemotron Elastic. Each bar annotated with the paper's own baseline in small text, and a footnote reading "different baselines; not directly comparable". Caption: "Every elastic paper reports a large saving against a different denominator."]

A Concrete Example

A team is shipping one assistant across a phone, a laptop and a cloud tier. They train one elastic parent and need to pick the laptop configuration. Here is the arithmetic they actually do.

Step 1: the parent. 32 layers, \(d_{\text{model}} = 3072\), SwiGLU FFN with hidden \(m = 12288\), grouped-query attention with 24 query heads and 8 key-value heads of dimension 128, tied embeddings over a 256k vocabulary.

• FFN per layer at full width: \(3 \times 3072 \times 12288 = 113.2\)M (gate, up, down).
• Attention per layer: \(3072^2\) for Q, \(3072 \times 1024\) each for K and V, \(3072^2\) for O, giving \(25.2\)M.
• Embeddings: \(256000 \times 3072 = 786\)M.
• Parent total: \(32 \times (113.2 + 25.2) + 786 = 5215\)M, call it 5.2B.

Step 2: what the granularities give you. FFN granularities \(\{m/8, m/4, m/2, m\}\) cost \(\{14.2, 28.3, 56.6, 113.2\}\)M per layer. Everything else is fixed at \(32 \times 25.2 + 786 = 1591\)M.

Step 3: the budget does not land on a rung. The laptop tier has room for about 2.8B parameters. Uniform \(m/4\) leaves 300M on the table; uniform \(m/2\) overshoots by 600M. Mix'n'Match solves for a blend. Let \(x\) layers run at \(m/2\) and \(32 - x\) at \(m/4\):

which gives \(905.9 + 28.3x = 1209\), so \(x = 10.7\), rounded to 11. Eleven layers at \(m/2\) and twenty-one at \(m/4\) come to \(622.8 + 594.5 = 1217\)M of FFN, for a total of 2809M.

Step 4: which eleven. Not arbitrary. The team sweeps candidate assignments against a held-out validation set, and the winner concentrates the wide layers in the middle third of the stack, where per-layer importance measures consistently rank highest and where depth-pruning work finds the least removable blocks. Early layers do local, low-rank work; the final layers are dominated by the unembedding. The middle is where the width is worth paying for.

Step 5: the latency claim. Decode at batch size one is bandwidth-bound, so estimate time per token as bytes moved over effective bandwidth. At int4, weights are roughly half a byte per parameter:

• 2808M params \(\approx\) 1.40 GB per token.
• 5215M params \(\approx\) 2.61 GB per token.

On a 2025-class laptop accelerator sustaining on the order of 100 GB/s of effective bandwidth, that is roughly 14 ms and 26 ms per token, so about 71 and 38 tokens per second. The elastic configuration is not a little faster; it is close to twice as fast, because the ratio of weights is what sets the ratio of decode times in this regime.

Step 6: the number that ruins the story. The KV cache is untouched by any of this. Per token, per layer: 8 KV heads \(\times\) 128 dims \(\times\) 2 tensors \(\times\) 2 bytes = 4096 bytes. Across 32 layers that is 128 KiB per token, so a 32,768-token context holds 4.29 GB of cache. The elastic choice moved 1.2 GB of weights. The context window is holding three times that, and no FFN granularity will touch it.

That is the moment the team's design meeting turns, correctly, from elasticity to cross-layer KV sharing and cache quantisation. Elasticity is the right lever for compute and for latency at short context. It is the wrong lever for a long-context memory problem, and the arithmetic says so in one line.

[IMAGE: Stacked area chart, x-axis context length from 0 to 128k tokens, y-axis gigabytes. Two stacked regions: int4 weights (flat at 1.40 GB for the elastic config, with a dashed line at 2.61 GB for the parent) and fp16 KV cache growing linearly at 128 KiB per token, crossing the weight line near 11k tokens. Caption: "Past roughly 11k tokens, the cache is the model's memory footprint and elasticity is arguing about the smaller number."]

Where It Breaks

Memory is not what you bought

This is the most common misreading, and it is worth being blunt. A 6B child of a 12B parent requires the 12B parent's weights to be loadable, because the child's weights are a slice of the parent's. Elasticity buys FLOPs per token and therefore latency and throughput. It does not reduce footprint unless you materialise the slice and ship it as its own artefact, and at that point you are storing N files again and have saved only on training.

The consequence for planning: elastic architectures are a training-cost and serving-latency technology. If your constraint is "the model must fit in 8 GB", elasticity alone does not solve it. Quantisation, pruning to a real smaller checkpoint, or offload do.

[IMAGE: Two memory bars for the same 12B elastic parent serving a 6B child. Left bar, "what people assume": 6B of weights resident. Right bar, "what is actually resident": the full 12B of weights, with the 6B active slice shaded and the inactive remainder hatched but still occupying the bar. A third narrow bar shows the FLOPs per token, where the 6B slice really is half. Caption: "Compute halves, memory does not move."]

Granularities compete for the same weights

Joint training across granularities means the \(m/8\) path and the \(m\) path write gradient to the same parameters. At small scale this regularises and helps; at large scale, where the model is not over-parameterised for its data, it is a tax on the largest configuration. Progressive shrinking, staged curricula and distillation from the full model into the sub-models all exist to manage this, and none of them makes it disappear. Expect the parent of an elastic family to be slightly worse than the same model trained alone, and check that specifically rather than assuming.

The evaluation surface explodes

\(1.8 \times 10^{19}\) extractable configurations is a wonderful marketing number and an operational problem. Mix'n'Match optimises a parameter or latency budget. It does not establish that the resulting model is calibrated, that its refusal behaviour is intact, that its bias profile matches the parent, or that a safety fine-tune transferred to that slice. Nothing in the nesting guarantees these properties are monotone in size, and there is no reason to expect them to be.

What teams do in practice is certify a small number of named configurations and refuse to support the rest. That is the right call, and it means the shipped product has a ladder of sizes again. The saving was real, it was in training; the promise of a continuous deployment surface mostly did not survive contact with a safety review.

[IMAGE: Funnel diagram. Top band, "extractable configurations: 1.8e19". Second band, "configurations evaluated for loss: ~40". Third band, "configurations passing safety and calibration review: 3". Bottom band, "configurations shipped and supported: 3", labelled "the ladder, restored". Caption: "Elasticity is combinatorial; certification is not."]

Sliced FFNs do not touch attention state

Covered in the worked example, and it generalises. The FFN is where most parameters and most per-token FLOPs live, and none of the KV cache. For a workload with 20k-token prompts and 300-token completions, the parameter axis is not the binding one, and an elastic model will disappoint relative to its headline numbers. Know your prefill-to-decode ratio before choosing which efficiency technique to invest in.

The kernel has to exist

An elastic configuration is a set of odd matmul shapes. A granularity vector that assigns different hidden widths to different layers produces shapes that may not be aligned to the tile sizes a fused kernel expects, and the penalty for falling off the fast path is easily larger than the FLOP saving. This is the hardware lottery applied to a deployment-time decision rather than an architecture: the fastest configuration under a FLOP budget and the fastest configuration in wall-clock are not the same configuration, and the second one is what you want. Mix'n'Match against measured latency, not against parameter count.

Elastic reasoning models add a control problem

When the budget varies within a request, as in Star Elastic's phase-dependent scheme, something has to decide when to switch. That decision is made on partial information, mid-generation, and getting it wrong costs quality on exactly the requests that needed the compute. This is new enough, as of 2026, that the failure modes are not well characterised. Treat published gains as promising rather than settled.

Alternative Designs

Two of these compose rather than compete. Quantisation is orthogonal to elasticity and is the first thing you do, not an alternative to it. MoE and nested elasticity are both conditional computation, and the 2026 work applies both at once, which is what makes group-aware elastification of Mamba and MoE layers a research topic rather than an engineering detail.

How It Is Used in Practice

The one place a general audience has already used an elastic model is a phone. Gemma 3n ships E2B and E4B, where E stands for effective: raw parameter counts are roughly 5B and 8B, and the models run in about 2 GB and 3 GB of accelerator memory (Google, 2025, Introducing Gemma 3n: the developer guide). Three techniques produce that gap, and only one of them is elasticity:

• MatFormer makes E2B a nested sub-network of E4B, so the smaller model is a selection of FFN granularities rather than a separate download.
• Per-Layer Embeddings move a large block of embedding parameters into system memory, computed on the CPU, so accuracy tracks the full parameter count while footprint tracks the resident subset. This is the technique doing the memory work, not MatFormer.
• KV cache sharing shares a middle layer's keys and values with the layers above it, in the style of YOCO, giving a reported 2x faster prefill than Gemma 3 4B.

Google also documents Mix'n'Match for Gemma 3n, so a developer can extract intermediate sizes between E2B and E4B rather than choosing between them. Whether anyone should do that in a shipped product runs straight into the evaluation problem above.

On the datacentre side, the 2025 and 2026 work is aimed squarely at reasoning models, where a family's cost is compounded by the reasoning post-training each member needs. Nemotron Elastic's stated motivation is that training a family, from scratch or by iterative compression, is prohibitive per member. Nesting the family means the reasoning curriculum runs once.

The engineering that follows is unglamorous and decisive. A serving stack has to represent a configuration as data rather than as a model identity, so that routing, autoscaling and billing all understand that two "models" share weights and a process. Observability has to attribute latency and quality per configuration. And a rollback has to be able to name a configuration, because "roll back to the 4B" is meaningless when the 4B is a view.

[IMAGE: Annotated architecture diagram of a serving deployment. One GPU holds a single set of parent weights; three request streams (phone-class, laptop-class, cloud-class) enter a gateway, hit a selector that emits granularity vectors, and share the same engine process. Callouts mark where per-configuration metrics are emitted and where the rollback boundary sits. Caption: "The operational unit is the configuration, not the checkpoint."]

Insights Worth Remembering

1. Elasticity moves cost out of training and leaves it in evaluation. The pretraining saving is the headline and it is genuine. The evaluation, safety certification and support burden per shipped size is unchanged, and that is usually the larger organisational cost.

2. A nested child is not a smaller model, it is a cheaper forward pass. Until you materialise and ship the slice, the parent's weights are the footprint. Any plan that says "we'll use the 2B on devices with 4 GB" needs to state which of those two things it means.

3. Prefixes, not subsets. The reason nesting works in memory is that every sub-network is a contiguous prefix of the parent's weight matrices. Arbitrary sub-networks would require separate storage and would forfeit the entire point.

4. Four granularities are a continuum, not a ladder. Per-layer selection turns \(|G|\) choices into \(|G|^{L}\) configurations. This is the single idea that separates MatFormer from slimmable networks, and it costs nothing at inference.

5. Small granularities regularise large ones, up to a point. MatFormer's 850M sub-models beat independently trained baselines; its 2.6B ones merely matched them. Expect the benefit to shrink and then invert as the parent stops being over-parameterised for its data.

6. Compression and elasticity solve adjacent problems, not the same one. Prune-and-distil gives you cheap independent checkpoints. Elastic training gives you serve-time switching from one checkpoint. Pick by whether your pain is training cost or deployment rigidity.

7. Mix'n'Match against latency, not parameters. A configuration chosen on parameter count can land on unaligned matmul shapes and be slower than a larger, better-shaped one. The budget that matters is milliseconds.

8. The KV cache is the other half of the bill, and elasticity does not pay it. In the worked example the cache passes the weights at about 11k tokens of context. Efficiency work that ignores which term dominates at your context length is optimising the wrong number.

Open Questions

Does nesting scale to frontier models? Measured results run to 12B parents. Whether the interference between granularities stays manageable at 100B or beyond, where models are much closer to compute-optimal and have less slack to give the small configurations, has not been demonstrated publicly. It is plausible that the regularisation benefit inverts well before that scale.

Can safety properties be made monotone in configuration? It is known that a safety fine-tune applied to the parent does not automatically hold for every slice, and it is not known whether training objectives can be designed to make refusal behaviour, calibration and bias vary predictably with budget. Without that, the combinatorial deployment surface is not usable in regulated settings, and the certified-ladder workaround is permanent rather than temporary.

What is the right granularity axis? MatFormer nests the FFN; Nemotron Elastic elastifies MLP, SSM groups and depth; YOCO-style designs restructure attention entirely. Nobody has published a controlled comparison of which axis gives the most deployable range per unit of quality lost, and the answer likely depends on whether prefill or decode dominates the workload.

Does within-request elasticity actually pay? Star Elastic's phase-dependent scheme reports improvements on the accuracy-speed Pareto frontier, which is evidence that it can. Whether a controller can reliably decide mid-generation when to change budget, and what the tail behaviour looks like when it decides wrongly on a hard problem, is open as of 2026.

Does elasticity survive quantisation? Every shipped on-device model is quantised, and low-bit quantisation is sensitive to outlier channels. Whether the prefix structure interacts badly with per-channel scaling, and whether a configuration that quantises cleanly at \(m/2\) still does at \(m/8\), is not something the elastic papers report. Anyone deploying this should measure it before assuming it composes.

Sources and Further Reading

1. Devvrit, Kudugunta, S., Kusupati, A., Dettmers, T., Chen, K., Dhillon, I., Tsvetkov, Y., Hajishirzi, H., Kakade, S., Farhadi, A., & Jain, P. (2023). "MatFormer: Nested Transformer for Elastic Inference." NeurIPS 2024. arXiv:2310.07707
2. Cai, R., Muralidharan, S., Heinrich, G., Yin, H., Wang, Z., Kautz, J., & Molchanov, P. (2024). "Flextron: Many-in-One Flexible Large Language Model." ICML 2024. arXiv:2406.10260
3. NVIDIA (2025). "Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs." arXiv:2511.16664
4. "Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control." (2026). ICML 2026. arXiv:2605.07182
5. Kusupati, A., Bhatt, G., Rege, A., Wallingford, M., Sinha, A., Ramanujan, V., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., & Farhadi, A. (2022). "Matryoshka Representation Learning." NeurIPS 2022. arXiv:2205.13147
6. Yu, J., Yang, L., Xu, N., Yang, J., & Huang, T. (2018). "Slimmable Neural Networks." ICLR 2019. arXiv:1812.08928
7. Cai, H., Gan, C., Wang, T., Zhang, Z., & Han, S. (2019). "Once-for-All: Train One Network and Specialize it for Efficient Deployment." ICLR 2020. arXiv:1908.09791
8. Muralidharan, S., Turuvekere Sreenivas, S., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., & Molchanov, P. (2024). "Compact Language Models via Pruning and Knowledge Distillation." arXiv:2407.14679
9. Grattafiori, A., et al. (Meta) (2024). "The Llama 3 Herd of Models." arXiv:2407.21783
10. Sun, Y., Dong, L., Zhu, Y., Huang, S., Wang, W., Ma, S., Zhang, Q., Wang, J., & Wei, F. (2024). "You Only Cache Once: Decoder-Decoder Architectures for Language Models." NeurIPS 2024. arXiv:2405.05254
11. Brandon, W., Mishra, M., Nrusimha, A., Panda, R., & Ragan-Kelley, J. (2024). "Reducing Transformer Key-Value Cache Size with Cross-Layer Attention." arXiv:2405.12981
12. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023). "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints." EMNLP 2023. arXiv:2305.13245
13. Liu, Z., Zhao, C., Iandola, F., Lai, C., Tian, Y., Fedorov, I., Xiong, Y., Chang, E., Shi, Y., Krishnamoorthi, R., Lai, L., & Chandra, V. (2024). "MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases." ICML 2024. arXiv:2402.14905
14. Google (2025). "Introducing Gemma 3n: The developer guide." Google Developers Blog
15. Wu, Y., Wu, H., & Tu, K. (2024). "A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference." NAACL 2025. arXiv:2410.14442
16. Hooker, S. (2020). "The Hardware Lottery." Communications of the ACM, 64(12), December 2021. arXiv:2009.06489

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.