Model Merging: Linear and SLERP

You can train two models - one specialised for reasoning and one for creative writing - and merge them into a single checkpoint that costs nothing extra at inference time. No additional training, no architectural changes, no ensembling overhead. That is the practical promise of model merging, and it is now routinely used to build competitive open-weight models on Hugging Face.

This concept covers the two most widely used interpolation strategies: plain linear (weighted average) and SLERP (Spherical Linear Interpolation). Both operate entirely in weight space, require no labelled data, and produce a checkpoint identical in shape to either parent.

The geometry of fine-tuned weight space

A pre-trained model occupies a point theta_base in a very high-dimensional weight space (billions of dimensions for a 7B model). Fine-tuning nudges that point toward a region of the manifold that is good for some target behaviour. The key empirical observation, formalised in the "Task Arithmetic" paper (Ilharco et al., ICLR 2023), is that the delta between fine-tuned and base weights, called a task vector, is a meaningful direction:

tau = theta_ft - theta_base

Adding or scaling task vectors tends to combine capabilities in a surprisingly linear way, at least when the fine-tuned models share the same base checkpoint and were not driven too far from it.

This linearity is not guaranteed by theory; it is an empirical regularity rooted in the fact that modern pre-training leaves the loss landscape locally quite flat around theta_base. Models fine-tuned from the same base therefore often sit in the same loss basin, connected by low-loss paths - the precondition that makes weight averaging sensible.

Linear merging (weighted average)

Given two fine-tuned models A and B, the linear merge is:

theta_merged = alpha * theta_A + (1 - alpha) * theta_B

where alpha in [0, 1] controls the blend. When alpha = 0.5 this is a simple arithmetic mean, sometimes called a "model soup" after the Wortsman et al. (2022) paper that popularised this recipe for ensembling multiple hyperparameter runs of the same task.

For a full layer-by-layer merge you apply this independently to every weight tensor: attention projections, MLP weights, layer norms, embeddings. Because the merged tensor has exactly the same shape as either parent, the model can load and run immediately.

Practical properties:

Searching alpha is cheap: you can sweep 10-20 values and evaluate on a held-out set in minutes. The Wortsman et al. model soups paper demonstrated that averaging models fine-tuned with different hyperparameter configurations consistently outperforms any individual model on CLIP and ViT benchmarks.

SLERP: interpolating on the hypersphere

Linear interpolation in Cartesian coordinates ignores the geometry of the weight space. When two weight vectors u and v are not collinear, the linear midpoint 0.5 * (u + v) has a shorter L2 norm than either endpoint - the interpolant "sags inward." For rotational quantities (quaternions, normalised activations) this causes distortions; for LLM weights it can shrink effective magnitudes in ways that degrade performance.

SLERP, originally designed for smooth quaternion animation (Shoemake, SIGGRAPH 1985), fixes this by travelling along the great circle connecting u and v on the hypersphere:

SLERP(u, v, t) = sin((1-t)*Omega) / sin(Omega) * u
               + sin(t*Omega)     / sin(Omega) * v

where Omega = arccos( dot(u_hat, v_hat) )
      u_hat = u / ||u||,  v_hat = v / ||v||

The result has the same magnitude as the endpoints (modulo the norm ratio), and the angular midpoint is equidistant from both in angle rather than in Euclidean distance.

In practice, SLERP is applied per weight tensor (or even per row/column of a large matrix), treating each as a vector on its own hypersphere. Implementations like mergekit (Arcee AI) handle this automatically.

When does SLERP help over linear? In high-dimensional spaces where the two weight tensors subtend a meaningful angle (say, over 10-15 degrees), SLERP tends to preserve the "energy" of each expert better. Practitioners report softer capability blends with fewer abrupt degradations at extreme t values. The difference is usually small for closely related models and more noticeable when merging models fine-tuned in quite different directions (e.g., a chat model and a coding model from the same base).

A minimal mergekit configuration

mergekit (github.com/arcee-ai/mergekit) exposes both methods through a YAML config:

# SLERP merge of two 7B models
merge_method: slerp
base_model: mistralai/Mistral-7B-v0.1
models:
  - model: model-A   # e.g. chat-tuned
    parameters:
      t: [0, 0.5, 1]   # per-layer schedule
  - model: model-B   # e.g. code-tuned
dtype: bfloat16

For a plain linear merge, change merge_method: linear and supply weight: values instead of t. The tool handles layer-by-layer dispatch, streaming to disk when VRAM is limited.

A typical sweep workflow: 1. Pick alpha (or t) from {0.3, 0.4, 0.5, 0.6, 0.7}. 2. Evaluate each merged checkpoint on a held-out task set (e.g., OpenLLM leaderboard tasks). 3. Pick the value that best satisfies your capability trade-off.

When it falls down

Loss basin mismatch. The linear path between two models is only low-loss when they sit in the same loss basin. If one model was fine-tuned for very many steps or with a large learning rate, its weights may have moved far enough that the interpolant crosses a loss barrier - performance at intermediate alpha can be worse than either endpoint. The "loss barrier" problem is well-documented in model merging literature; TIES-Merging (Yadav et al., NeurIPS 2023) was specifically motivated by this interference effect.

Sign conflicts destroy information. When corresponding parameters in models A and B have opposite signs with similar magnitudes, their average is near zero - effectively zeroing out features that both models used but in opposite polarity. TIES-Merging addresses this by resolving sign conflicts before averaging; plain linear and SLERP do not.

Different base checkpoints cannot be merged. SLERP and linear both assume that the two models share the same architectural initialisation. Merging a Mistral-7B fine-tune with a Llama-3-8B fine-tune produces noise, not a useful model. The models need not be identical checkpoints but must have been trained from compatible weight initialisations.

SLERP is undefined when vectors are parallel or anti-parallel. When Omega is close to 0 (vectors nearly identical) or pi (vectors opposite), sin(Omega) approaches zero and the formula is numerically unstable. Implementations fall back to linear interpolation in this case, but the fall-back is silent - worth checking in low-level implementations.

Catastrophic forgetting can be hidden. A merged model may score well on benchmarks while quietly losing specific safety tuning or instruction-following nuance. Merging is not a substitute for evaluating alignment properties directly.

Further reading

• Ilharco et al., "Editing Models with Task Arithmetic," ICLR 2023. https://arxiv.org/abs/2212.04089 - Foundational paper establishing task vectors and their additive properties.
• Wortsman et al., "Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time," ICML 2022. https://arxiv.org/abs/2203.05482 - Empirical case for weight averaging, with loss-flatness analysis.
• Yadav et al., "TIES-Merging: Resolving Interference When Merging Models," NeurIPS 2023. https://arxiv.org/abs/2306.01708 - Documents sign-conflict and magnitude-drop failure modes; motivates going beyond linear.
• Hugging Face blog: "Merge Large Language Models with mergekit" (2024). https://huggingface.co/blog/mlabonne/merge-models - Practical walkthrough of linear, SLERP, TIES, and DARE with mergekit config examples.