Model Merging: Why Averaging Weights Works, and Where the Free Lunch Ends
Three 7B models that each scored under 30% on Japanese maths were averaged into one that scored 52%. No gradient was computed. Weight-space arithmetic is the cheapest capability gain in the field and the easiest one to fool yourself with, and both facts come from the same property of the loss landscape.
Take three open 7B models: a Japanese general-purpose model and two English mathematical reasoning models. Each scores below 30% on MGSM-JA, a set of Japanese mathematical word problems. Average their weights under a search over the mixing coefficients and the result scores 52.0%; add a layer-permutation search on top and it reaches 55.2% (Akiba et al., 2025, Evolutionary optimization of model merging recipes, Nature Machine Intelligence 7, 195–204, arXiv:2403.13187). No gradient was computed. No training data was touched. The operation was elementwise arithmetic over three tensors that happened to be lying around on a model hub.
This should be alarming as well as impressive. A procedure that produces a 22-point jump for the cost of a torch.add is either exploiting something deep about how fine-tuning works, or it is exploiting something shallow about how we evaluate. It is mostly the former, and the part that is the latter is where teams get hurt.
Why this matters: Merging is now the default way to combine specialised models, ship capabilities without a training run, and recover from a fine-tune that broke something. It is also the operation most likely to silently delete safety behaviour, because the mechanism that fuses skills and the mechanism that erases refusals are the same mechanism.
TL;DR
- Weight averaging works only between models that share a pretrained initialisation and never left its basin. Two models trained independently from scratch average into noise, and the loss barrier between them is the reason.
- Fine-tuning moves weights very little. Supervised fine-tuning deltas typically sit within 0.002 in magnitude, and 90% of them, sometimes 99%, can be zeroed and the rest rescaled with no measurable capability loss (Yu et al., 2024, arXiv:2311.03099). Merging is arithmetic on a nearly empty tensor.
- The enemy is not averaging, it is interference: redundant parameters dilute the signal, and parameters whose sign disagrees across models cancel. TIES-Merging attacks both directly and beats naive averaging because of it (Yadav et al., 2023, arXiv:2306.01708).
- Naive averaging of \(N\) task vectors divides every task's own update by \(N\) before interference is even considered. The scaling coefficient is not a tuning nicety; it is a correction for a systematic attenuation.
- Returns diminish sharply in the number of experts and improve with base model size. Across 10,866 merged models from 0.5B to 72B, most of the gain arrives from the first few experts and the differences between merging methods shrink at scale (Model Merging Scaling Laws in Large Language Models, 2025, arXiv:2509.24244).
- Safety is the asymmetric casualty. Merging an unaligned skill model into an aligned one degrades refusal behaviour, and standard capability benchmarks will not show it (Hammoud et al., 2024, arXiv:2406.14563).
- Merging costs nothing at inference. This is its real advantage over ensembling, and the reason it displaced ensembles in production rather than in the literature.
At a Glance
flowchart LR
B["Pretrained base θ0"] --> F1["Fine-tune: maths"]
B --> F2["Fine-tune: code"]
B --> F3["Fine-tune: Japanese"]
F1 --> T["Task vectors τ = θ − θ0"]
F2 --> T
F3 --> T
T --> R["Resolve interference: trim, elect sign"]
R --> S["Scale by λ and add to θ0"]
S --> M["One model, one forward pass"]
classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
class B blue
class F1,F2,F3 purple
class T,R,S amber
class M tealThe whole field is that diagram plus arguments about the box labelled "resolve interference".
Before Weight Arithmetic
Averaging weights is older than the current interest in it, and it started as a regularisation trick rather than a composition trick. Stochastic Weight Averaging averaged points along a single SGD trajectory and found flatter, better-generalising solutions than the trajectory's endpoint (Izmailov et al., 2018, Averaging Weights Leads to Wider Optima and Better Generalization, UAI, arXiv:1803.05407). Nobody at that point was averaging different models, because the obvious experiment, averaging two independently trained networks, produces a model at chance accuracy.
The theory that explained why arrived with instability analysis. Frankle and colleagues asked whether a network trained twice under different SGD noise lands in the same linearly connected region, and found that standard vision models become stable to that noise early in training, after which the outcome is confined to one linearly connected basin (Frankle, Dziugaite, Roy and Carbin, 2020, Linear Mode Connectivity and the Lottery Ticket Hypothesis, ICML, arXiv:1912.05671). Stability is the licence to average. Before the stability point, two runs diverge into different basins and the straight line between them crosses a high-loss barrier; after it, the line stays low and the midpoint is a valid model.
That is the entire foundation of model merging, and it has a sharp corollary: everything you merge must descend from a common ancestor, past the point of stability. Git Re-Basin showed the constraint is about coordinates rather than fate, by permuting one network's hidden units into alignment with another's so that the two land in an approximately convex shared basin (Ainsworth, Hayase and Srinivasa, 2023, Git Re-Basin: Merging Models modulo Permutation Symmetries, ICLR, arXiv:2209.04836). Elegant, and in practice almost nobody needs it, because everyone is fine-tuning from the same public checkpoint anyway.
timeline
title From regularisation trick to composition primitive
2018 : SWA averages along one SGD trajectory for flatter optima
2020 : Linear mode connectivity explains when averaging is legal
2021 : Fisher-weighted averaging weights each parameter by curvature
2022 : Model soups reach 90.94 percent on ImageNet by averaging a sweep
: Git Re-Basin aligns units by permutation before merging
: Task arithmetic defines the task vector and negates it
2023 : TIES-Merging trims, elects signs and resolves interference
: DARE drops 90 percent of deltas and rescales the rest
2024 : mergekit standardises the recipes; evolutionary search tunes them
2025 : MergeBench and scaling-law studies make the returns predictableTwo papers in 2022 turned averaging into a capability tool. Model soups averaged the weights of many fine-tuning runs from one hyperparameter sweep, rather than selecting the best run, and pushed a JFT-pretrained ViT-G to 90.94% top-1 on ImageNet with no additional inference or memory cost (Wortsman et al., 2022, Model soups, ICML, arXiv:2203.05482). The framing mattered as much as the number: a hyperparameter sweep usually throws away every run but one, and the discarded runs contain information.
Task arithmetic supplied the vocabulary everyone now uses. Define the task vector \(\tau = \theta_{\text{ft}} - \theta_0\), the difference between a fine-tuned model and its base. Ilharco and colleagues showed these vectors compose: adding them builds multitask models, and negating one removes the corresponding behaviour while leaving control tasks largely intact (Ilharco et al., 2023, Editing Models with Task Arithmetic, ICLR, arXiv:2212.04089). Negation is the striking half. A capability you can subtract is a capability that lives in a direction, not in the whole network.
[IMAGE: Two-panel loss-landscape illustration. Left panel: two independently initialised models as points in separate basins, with a high loss barrier along the straight line between them and a plot of loss along that line peaking in the middle. Right panel: one pretrained base with three fine-tunes branching into a single wide basin, the straight lines between them staying at low loss. Caption: "Averaging is only defined inside a basin. Shared pretraining is what puts the models there."]
How Merging Actually Works
The delta is almost empty
Start with the fact that makes everything else cheap. Fine-tuning barely moves the weights. Yu and colleagues measured the delta parameters of supervised fine-tuned models and found their values typically fall within 0.002 in magnitude, with extreme redundancy across coordinates, and that randomly zeroing 90% of them, or even 99%, then rescaling the survivors, leaves the model's abilities intact (Yu et al., 2024, Language Models are Super Mario, arXiv:2311.03099).
The rescaling is the interesting part. If you drop each delta independently with probability \(p\) and multiply the survivors by \(1/(1-p)\), the expected value of the delta at every coordinate is preserved:
DARE is dropout applied to a finished model rather than during training, and it works for the same reason dropout works: the network's function is robust to unbiased, high-variance perturbations of individual weights. What it buys is not compression. It is disjointness. If each task vector is 90% zeros, two task vectors overlap on roughly 1% of coordinates instead of 100%, and most of the interference that merging fights simply never occurs.
[IMAGE: Histogram of delta-parameter magnitudes for one weight matrix of a supervised fine-tuned model, log-scaled y-axis, with a sharp spike at zero and a narrow tail ending before 0.002, and a vertical marker at the 90th percentile of magnitude annotated "everything left of here can be dropped and rescaled". Caption: "Fine-tuning barely moves the weights, and almost none of the movement is load-bearing."]
Interference has two distinct causes
Naive averaging of \(N\) fine-tunes computes \(\theta_0 + \frac{1}{N}\sum_i \tau_i\). Two things go wrong, and conflating them leads to the wrong fix.
The first is dilution. A coordinate where only the maths model has a meaningful update gets that update divided by \(N\). Nothing conflicted; the signal was simply averaged against zeros. This is why task arithmetic uses \(\theta_0 + \lambda \sum_i \tau_i\) with a tuned \(\lambda\) rather than a fixed \(1/N\): the correct scale is an empirical question, usually landing somewhere between \(1/N\) and 1, and tuning it is consistently among the highest-value knobs in the whole procedure.
The second is sign conflict. A coordinate where the maths model wants \(+0.021\) and the Japanese model wants \(-0.025\) is a genuine disagreement, and averaging resolves it by splitting the difference, which satisfies neither. TIES-Merging identifies these two causes explicitly and addresses each: trim each task vector to its top-magnitude entries, zeroing the redundant ones; elect a sign per coordinate by comparing the total magnitude of positive against negative proposals; merge by taking the mean only over the entries that agree with the elected sign (Yadav et al., 2023, TIES-Merging, NeurIPS, arXiv:2306.01708).
The disjoint mean is the step that does the work. Averaging over agreeing entries only, rather than over all \(N\) models, means a coordinate that two models care about and one does not keeps roughly the magnitude those two asked for.
[IMAGE: Three-row heatmap over 60 parameter coordinates, one row per task vector, cells coloured blue for a positive delta, rose for negative, grey for near-zero. Most cells are grey; a handful of columns show two blue and one rose, annotated "sign conflict: averaging splits the difference", and others show one coloured cell among two grey, annotated "dilution: divided by N for no reason". Caption: "Two different failures that look identical in an averaged tensor."]
Weighting by curvature, and by geometry
Two other families are worth knowing because they answer different questions.
Fisher-weighted averaging asks which parameters each model actually cares about. Treating each fine-tuned model as a Gaussian posterior whose precision is its Fisher information, the merge that maximises the joint likelihood weights each parameter by that model's Fisher diagonal, so a parameter a model is sensitive to dominates one it is indifferent to (Matena and Raffel, 2022, Merging Models with Fisher-Weighted Averaging, NeurIPS, arXiv:2111.09832). Plain averaging is the special case where every posterior is isotropic. The cost is estimating the Fisher diagonal, which needs data and a backward pass, so it sits awkwardly between "free" and "training".
Spherical linear interpolation, SLERP, asks a geometric question instead: interpolating two weight vectors along a straight line shortens the result, because the chord of a sphere is shorter than its arc. SLERP interpolates along the arc, preserving norm. It is the default for two-model merges in practice and does not generalise cleanly past two.
Seeing It in Motion
flowchart TB
Q{"How many models, and do they share a base?"}
Q -->|"Different pretraining"| X["Merging is not defined; distil or route instead"]
Q -->|"Two, same base"| S["SLERP or linear with tuned λ"]
Q -->|"Many, same base"| D{"Are the fine-tunes on conflicting objectives?"}
D -->|"Mostly disjoint skills"| L["Linear or DARE, tune λ"]
D -->|"Overlapping or conflicting"| T["TIES: trim, elect sign, disjoint mean"]
D -->|"Have data and a backward pass"| F["Fisher-weighted averaging"]
T --> E["Evaluate: per-task, forgetting, safety"]
L --> E
S --> E
F --> E
E -->|"Safety regressed"| G["Re-merge with alignment-preserving weights"]
classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
class Q,D slate
class S,L,T,F purple
class E teal
class X,G roseThe decision that matters most is the first one, and it is the one people skip. Two models with different tokenisers, different pretraining corpora, or different architectures are not mergeable in this sense, however similar their parameter shapes. The shapes lining up is a coincidence of configuration, not evidence of a shared basin.
sequenceDiagram
participant E as Engineer
participant M as mergekit
participant C as Capability evals
participant S as Safety evals
E->>M: Config: base, 4 task vectors, TIES, λ=0.7
M-->>E: Merged checkpoint (minutes, CPU-only)
E->>C: Per-task benchmarks plus held-out general set
C-->>E: Target tasks up, general set roughly flat
E->>S: Refusal and jailbreak suite
S-->>E: Harmful-response rate up
Note over E,S: Capability evals passed; the regression is invisible to them
E->>M: Re-merge, higher weight on the aligned model
M-->>E: Candidate 2Note what the sequence costs. The merge itself is minutes on a CPU. The evaluation is hours on accelerators, and it is where the entire budget of a merging programme goes. A team that treats merging as cheap has usually forgotten to price the second half.
[IMAGE: Line chart, x-axis "number of expert models merged" from 1 to 16, y-axis "average task score". Three curves for base model sizes 1B, 8B and 64B, each rising steeply from 1 to 4 experts and flattening after 8, with the larger models both higher and flatter, and a shaded band around each curve narrowing as expert count grows. Caption: "Most of the gain arrives from the first few experts, and variance contracts as more are added."]
Watch It Run
By the Numbers
| Quantity | Setting | Value | Source |
|---|---|---|---|
| Greedy soup top-1 accuracy | ViT-G pretrained on JFT, ImageNet | 90.94% | Wortsman et al., 2022 |
| Typical SFT delta magnitude | Per-parameter, supervised fine-tuned LMs | within 0.002 | Yu et al., 2024 |
| Delta parameters droppable | With \(1/(1-p)\) rescaling, no measured capability loss | 90%, up to 99% | Yu et al., 2024 |
| MGSM-JA, individual source models | Three 7B models, Japanese maths word problems | below 30% | Akiba et al., 2025 |
| MGSM-JA, evolved parameter-space merge | Same three models, searched mixing coefficients | 52.0% | Akiba et al., 2025 |
| MGSM-JA, parameter plus layer search | Same three models, hybrid PS and DFS merge | 55.2% | Akiba et al., 2025 |
| Merged models in the scaling study | 0.5B–72B bases, nine domains, four methods | 10,866 | arXiv:2509.24244, 2025 |
| MergeBench coverage | Llama and Gemma families, five domains | 2B–9B, 8 methods | He et al., 2025 |
| Model-size range in the PaLM-2 study | Experts merged from 1B to 64B bases | 1B–64B | Yadav et al., 2024 |
| Inference cost of a merge vs an \(N\)-model ensemble | Same architecture, one forward pass | 1x vs \(N\)x | structural |
Sources: model soups (Wortsman et al., 2022); DARE (Yu et al., 2024); evolutionary merging (Akiba et al., 2025); merging scaling laws (arXiv:2509.24244); MergeBench (He et al., 2025); merging at scale (Yadav et al., 2024). The MGSM-JA source-model figure is reported as a range below 30% rather than three individual scores. The final row is arithmetic, not a measurement.
A Concrete Example
Merge three task vectors over a five-coordinate slice of one weight matrix. Real tensors have billions of coordinates; the procedure is identical and this one fits on paper.
Step 1. The task vectors. Each is \(\theta_{\text{ft}} - \theta_0\) for one fine-tune.
| Coordinate | \(\tau_{\text{maths}}\) | \(\tau_{\text{code}}\) | \(\tau_{\text{ja}}\) |
|---|---|---|---|
| 1 | +0.021 | +0.018 | −0.025 |
| 2 | −0.004 | +0.011 | +0.003 |
| 3 | +0.009 | −0.012 | +0.010 |
| 4 | +0.002 | +0.001 | −0.001 |
| 5 | −0.030 | +0.026 | +0.022 |
Step 2. What naive averaging produces. Sum each row and divide by three:
- Coordinate 1: \((0.021 + 0.018 - 0.025)/3 = 0.0047\)
- Coordinate 2: \((-0.004 + 0.011 + 0.003)/3 = 0.0033\)
- Coordinate 3: \((0.009 - 0.012 + 0.010)/3 = 0.0023\)
- Coordinate 4: \((0.002 + 0.001 - 0.001)/3 = 0.0007\)
- Coordinate 5: \((-0.030 + 0.026 + 0.022)/3 = 0.0060\)
Look at coordinate 1. Two models asked for roughly \(+0.02\) and one asked for \(-0.025\); the merge delivers \(+0.0047\), under a quarter of what the majority wanted. Coordinate 5 is worse: the maths model wanted \(-0.030\), the other two wanted about \(+0.024\), and the result is \(+0.006\), a value none of the three would recognise. Meanwhile coordinate 4, which nobody cared about, survives with a non-zero update.
Step 3. Trim. Keep the top 40% of each task vector by magnitude, which here is two coordinates each.
| Coordinate | \(\tau_{\text{maths}}\) | \(\tau_{\text{code}}\) | \(\tau_{\text{ja}}\) |
|---|---|---|---|
| 1 | +0.021 | +0.018 | −0.025 |
| 2 | 0 | 0 | 0 |
| 3 | 0 | 0 | 0 |
| 4 | 0 | 0 | 0 |
| 5 | −0.030 | +0.026 | +0.022 |
Coordinates 2 through 4 are gone. Three of five coordinates were carrying no decision.
Step 4. Elect a sign. Per surviving coordinate, compare total positive mass against total negative mass.
- Coordinate 1: positive \(0.021 + 0.018 = 0.039\); negative \(0.025\). Elected sign: \(+\).
- Coordinate 5: positive \(0.026 + 0.022 = 0.048\); negative \(0.030\). Elected sign: \(+\).
Note what just happened at coordinate 5. The maths model's largest single update in this slice lost the vote and is discarded outright, rather than being partially honoured. Merging makes a decision where averaging made a compromise.
Step 5. Disjoint mean. Average only the entries matching the elected sign.
- Coordinate 1: \((0.021 + 0.018)/2 = 0.0195\)
- Coordinate 5: \((0.026 + 0.022)/2 = 0.0240\)
Step 6. Scale and apply. With \(\lambda = 1\), the merged delta is \([+0.0195,\ 0,\ 0,\ 0,\ +0.0240]\), against naive averaging's \([+0.0047,\ +0.0033,\ +0.0023,\ +0.0007,\ +0.0060]\). The surviving updates are roughly four times larger and the noise coordinates are exactly zero. That ratio of about \(N/2\) on agreeing coordinates is the general shape of the improvement, and it is why the scaling coefficient and the interference resolution are really the same correction applied at different granularities.
[IMAGE: Grouped bar chart over the five coordinates of the worked example. For each coordinate, three light bars showing the individual task-vector values, one bar for the naive average, and one bar for the TIES result, with the TIES bars at coordinates 2 to 4 flat at zero and the bars at coordinates 1 and 5 roughly four times the naive average. Caption: "Averaging spreads a small update everywhere; interference resolution concentrates a large one where the models agreed."]
Run DARE before this and the picture changes again: at \(p = 0.9\) each task vector is 90% zeros before trimming, the three vectors collide on a tenth as many coordinates, and the sign election has far fewer contested votes to settle.
Where It Breaks
Safety is not a capability and does not merge like one
This is the failure that has cost teams the most. Merging a capable but unaligned skill model into an aligned model degrades the aligned model's refusal behaviour, and the harmful-response rate of the merge can rise sharply (Hammoud et al., 2024, Model Merging and Safety Alignment: One Bad Model Spoils the Bunch, arXiv:2406.14563).
The mechanism is not mysterious. Refusal behaviour is concentrated: a comparatively small, coherent set of directions, often expressed in the first few generated tokens, carries most of it. A skill fine-tune that never trained on refusals has effectively random deltas in those directions, and the merge averages a deliberate signal against noise, attenuating it. What makes this worse than an ordinary capability regression is that nothing in the merge's provenance flags it. You merged a maths model; you did not merge a jailbreak.
[IMAGE: Grouped bar chart with four groups on the x-axis: aligned parent, skill parent, naive merge, alignment-weighted merge. Two bars per group, one for target-task score in teal and one for harmful-response rate in rose. The naive merge shows the highest teal bar and a rose bar close to the unaligned skill parent's. Caption: "The capability chart and the safety chart move in opposite directions, and only one of them is usually plotted."]
Two consequences follow. Safety evaluation is not optional post-merge even when every parent was aligned, and the merge weights that maximise task scores are frequently not the weights that preserve alignment, so the two objectives have to be traded explicitly rather than discovered.
The evaluation illusion
Merged models are unusually good at looking better than they are, for a structural reason: the merge is tuned on the benchmarks. Searching \(\lambda\), per-layer weights, or an evolutionary recipe against a target metric is hyperparameter optimisation with a very small number of effective parameters and a very large number of evaluations. That is a recipe for fitting the evaluation set.
The defences are ordinary and frequently skipped. Hold out a set the merge search never sees. Report general-capability scores alongside target-task scores, because merging trades against them. Measure forgetting explicitly against the base model, which is one of the three axes MergeBench was built to standardise, alongside multi-task performance and runtime efficiency (He et al., 2025, MergeBench, arXiv:2505.10833).
Diminishing returns are steep and predictable
The instinct after a successful two-model merge is to merge eight. The data says most of the gain was already collected. Across 10,866 merged models spanning 0.5B to 72B base sizes, nine domains and four methods, gains follow a power law in expert count with a size-dependent floor: most of the improvement comes from the first few experts, variance contracts as more are added, and the differences between Average, task arithmetic, TIES and DARE shrink as models get larger (arXiv:2509.24244).
That last point deserves emphasis because it inverts the usual reading of the method literature. Method choice matters most exactly where it is easiest to study, at small scale, and matters least where the money is.
Base model quality dominates method choice
Merging experts built on an instruction-tuned base consistently beats merging experts built on the raw pretrained base, and larger models merge more easily (Yadav et al., 2024, What Matters for Model Merging at Scale?, arXiv:2410.03617). MergeBench reached the same conclusion independently on Llama and Gemma at 2B to 9B.
The practical reading: if a merge underperforms, the first thing to change is the base, not the algorithm. A stronger base means the fine-tunes start from a better-conditioned region and their task vectors are smaller relative to the shared weights, which is precisely the regime in which linearity holds best.
The silent prerequisites
A merge will run, and produce a plausible checkpoint, under conditions where it is meaningless. Different tokenisers with coincidentally similar vocabulary sizes. One parent that was continued-pretrained far enough to leave the basin. A parent whose embedding matrix was resized. Models whose parameter names match but whose attention layouts differ in how heads are packed. None of these raise an exception; all of them produce a model that loads, generates fluent text, and is worse than either parent in ways that show up three benchmarks later.
Alternative Designs
| Design | How it works | Key advantage | Key limitation | Best when |
|---|---|---|---|---|
| Weight merging | Elementwise arithmetic on parameters from a shared base | Free at inference, no data, minutes on CPU | Requires shared base; deletes disagreement | Combining specialised fine-tunes of one model |
| Output ensembling | Run \(N\) models, combine logits or votes | No basin requirement, reliably beats the parts | \(N\) times the compute and memory at every request | Offline scoring where accuracy dominates cost |
| Multitask fine-tuning | One training run over all task data | Learns cross-task structure, no interference heuristics | Needs all data simultaneously; retraining for each new task | Data is available and centrally held |
| MoE upcycling | Convert fine-tunes into experts with a learned router | Keeps specialists intact, routes per token | Total memory is the sum of experts; router needs training | Skills genuinely conflict and must stay separate |
| Multi-adapter serving | Keep LoRA adapters separate, batch across them | Per-request specialisation, adapters stay auditable | Serving complexity; limited to adapter-sized deltas | Many tenants, each wanting their own behaviour |
| Distillation | Train one student on outputs of several teachers | No architectural constraints between teachers | A real training run, with data and compute | Teachers differ in architecture or tokeniser |
The honest comparison is against ensembling, because ensembling almost always produces a better model. Model soups analytically relate weight averaging to logit ensembling through the flatness of the loss surface and the confidence of the predictions, and in the regime where the approximation is good the two are close; where it is bad, ensembling wins. Merging is chosen anyway, because an \(N\)-times inference bill is not a rounding error and a torch.add is.
How It Is Used in Practice
mergekit is the standard implementation, integrated with the Hugging Face model hub and library, and it turned merging from a research script into a YAML file (Goddard et al., 2024, Arcee's MergeKit, arXiv:2403.13257). Its practical contribution is unglamorous and large: merging on CPU with tensors streamed from disk, so a 70B merge does not need 70B of accelerator memory, which is what made the technique accessible to people without clusters.
Three deployment patterns recur.
Checkpoint averaging inside a training run. The oldest and least controversial use: average the last several checkpoints, or the best runs from a sweep, before shipping. Costs nothing, usually helps slightly, and carries almost no risk because every input came from one training trajectory.
Skill composition across teams. An organisation with separate fine-tunes for retrieval, code and a domain vertical merges them rather than running a joint training job that would require pooling the data. This is merging's strongest argument, and it is an organisational argument rather than a technical one: it permits decentralised model development, with each team owning its own fine-tune.
Search over recipes. Treat the mixing coefficients, per-layer weights and even layer ordering as a search space and optimise them with an evolutionary algorithm, as in the EvoLLM-JP result that opened this article. The published recipe searched both the parameter space and the data-flow space, with the combination outperforming either alone. This is also where the evaluation-fitting risk is highest, since the search is explicitly maximising a benchmark.
[IMAGE: Annotated mergekit YAML config with callouts. Highlight the base_model field with the note "must be the common ancestor, not the best model", the merge_method field with "TIES if the fine-tunes conflict, SLERP for two", the per-model weight and density fields with "the two knobs that actually move results", and a final callout on an absent field reading "no field here declares which parent was safety-aligned". Caption: "The config is four decisions, and the one that causes incidents is not in the file."]
Insights Worth Remembering
-
Merging is arithmetic in a basin, not composition of models. Every property of the technique, the shared-base requirement, the sensitivity to the scaling coefficient, the failure on independently trained networks, follows from linear mode connectivity. If you remember one thing, remember that averaging is only defined where the straight line between two models stays at low loss.
-
The delta, not the model, is the object being merged. Fine-tuning deltas are tiny and mostly redundant, which is why 90% of them can be thrown away. Thinking in task vectors rather than in checkpoints makes negation, scaling and sparsification obvious operations rather than tricks.
-
Averaging deletes disagreement; it does not resolve it. The worked example shows a coordinate where the majority wanted \(+0.02\) and the merge delivered \(+0.005\). Interference-aware methods win by making a decision instead of a compromise, and the cost of that decision is that the losing model's update is discarded entirely.
-
The scaling coefficient is a correction, not a hyperparameter. Dividing by \(N\) attenuates every task's own update \(N\)-fold before conflict is considered. Tuning \(\lambda\) is not fine-tuning the merge; it is undoing a systematic error that naive averaging introduces.
-
Method choice matters less than base quality, and less at scale. Stronger and instruction-tuned bases merge better, larger models merge better, and the gap between Average, task arithmetic, TIES and DARE narrows as models grow. Change the base before changing the algorithm.
-
Safety regressions are invisible to capability evaluation. An aligned parent does not confer alignment on the merge, and no capability benchmark will tell you otherwise. A safety suite post-merge is as mandatory as a test suite post-refactor, and for the same reason: the operation was mechanical and did not know what it was touching.
-
Merging's advantage over ensembling is economic, not statistical. An ensemble is generally the better model. A merge is the better product, because it costs one forward pass. Anyone comparing them on accuracy alone has missed the argument.
Open Questions
Why does interference resolution work as well as it does? TIES and DARE are measured to help, consistently and across scales. The explanation, that fine-tuning deltas are redundant and sparsifiable, is well supported empirically by the DARE measurements. What is not established is a predictive theory: given two task vectors, nobody can currently say in advance how much they will interfere without merging them and measuring.
What is the right unit of merging? Whole-model merging treats every parameter identically. Layer-wise and per-module weights measurably help, and the evolutionary work searched the data-flow path as well as the parameter space. Whether there is a principled decomposition, by circuit, by module function, or by curvature, that beats searching, is open.
Can alignment be made a first-class constraint? Current practice detects safety loss after merging and then re-merges. Selective, layer-wise approaches that protect alignment-critical layers report better safety-utility trade-offs than indiscriminate merging (Djuhera et al., 2025, SafeMERGE, arXiv:2503.17239), but whether alignment can be guaranteed to survive an arbitrary merge, rather than empirically checked each time, is unresolved.
How far does the linearity assumption actually extend? The basin argument is established for fine-tuning from a shared checkpoint. It is not established for models that have undergone substantial continued pretraining, long RL post-training, or architecture surgery. Practitioners merge such models anyway. The boundary is currently found by hitting it.
Does merging remain useful as models get larger? The scaling data cuts both ways: larger models merge more easily, which favours merging, and method differences vanish at scale, which suggests the technique is converging on plain weighted averaging plus a tuned coefficient. That would be a good outcome for practitioners and a quiet ending for a research area.
Sources and Further Reading
- Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., & Wilson, A. G. (2018). "Averaging Weights Leads to Wider Optima and Better Generalization." UAI 2018. arXiv:1803.05407
- Frankle, J., Dziugaite, G. K., Roy, D. M., & Carbin, M. (2020). "Linear Mode Connectivity and the Lottery Ticket Hypothesis." ICML 2020. arXiv:1912.05671
- Matena, M., & Raffel, C. (2022). "Merging Models with Fisher-Weighted Averaging." NeurIPS 2022, 17703–17716. arXiv:2111.09832
- Wortsman, M., Ilharco, G., Gadre, S. Y., et al. (2022). "Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time." ICML 2022, PMLR 162. arXiv:2203.05482
- Ainsworth, S. K., Hayase, J., & Srinivasa, S. (2023). "Git Re-Basin: Merging Models modulo Permutation Symmetries." ICLR 2023. arXiv:2209.04836
- Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., & Farhadi, A. (2023). "Editing Models with Task Arithmetic." ICLR 2023. arXiv:2212.04089
- Yadav, P., Tam, D., Choshen, L., Raffel, C., & Bansal, M. (2023). "TIES-Merging: Resolving Interference When Merging Models." NeurIPS 2023. arXiv:2306.01708
- Yu, L., Yu, B., Yu, H., Huang, F., & Li, Y. (2024). "Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch." arXiv:2311.03099
- Goddard, C., Siriwardhana, S., Ehghaghi, M., Meyers, L., Karpukhin, V., Benedict, B., McQuade, M., & Solawetz, J. (2024). "Arcee's MergeKit: A Toolkit for Merging Large Language Models." arXiv:2403.13257
- Akiba, T., Shing, M., Tang, Y., Sun, Q., & Ha, D. (2025). "Evolutionary optimization of model merging recipes." Nature Machine Intelligence, 7, 195–204. arXiv:2403.13187
- Hammoud, H. A. A. K., Michieli, U., Pizzati, F., et al. (2024). "Model Merging and Safety Alignment: One Bad Model Spoils the Bunch." arXiv:2406.14563
- Yadav, P., Vu, T., Lai, J., et al. (2024). "What Matters for Model Merging at Scale?" arXiv:2410.03617
- He, Y., Zeng, S., et al. (2025). "MergeBench: A Benchmark for Merging Domain-Specialized LLMs." arXiv:2505.10833
- Djuhera, A., Kadhe, S. R., Ahmed, F., Zawad, S., & Boche, H. (2025). "SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging." arXiv:2503.17239
- "Model Merging Scaling Laws in Large Language Models." (2025). arXiv:2509.24244
- mergekit source and documentation. github.com/arcee-ai/mergekit
Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.