Composing and Stacking Adapters

Suppose you have already fine-tuned a 7B model with one LoRA adapter for Spanish translation and a second for legal summarisation. A third client needs a Spanish-language legal summariser. The naive answer is to train from scratch. The more interesting answer is to ask whether those two adapters can be combined into something useful without touching the base model again.

That question is what adapter composition is about. It is harder than it sounds, because LoRA and bottleneck adapters are not simply additive in the parameter space, and the ways they can interfere are subtle.

What "composition" actually means

The term covers at least three distinct operations, and conflating them leads to confusion:

Each carries different assumptions about whether the adapters were trained on compatible objectives, whether their rank subspaces overlap, and whether the base model can tolerate the combined perturbation.

Linear interpolation: the simplest case

For LoRA, the adapter's effect on a weight matrix W is:

Delta W = (alpha / r) * B @ A

If you have two adapters with deltas Delta_1 and Delta_2, the linearly interpolated model uses:

W_eff = W + lambda_1 * Delta_1 + lambda_2 * Delta_2

This is what the PEFT library's add_weighted_adapter method implements, with combination_type="linear". The result lives in the same weight tensor; no adapter scaffolding survives at inference time.

When does this work? When the two tasks are genuinely compatible - the adapters push the model in related directions, and their effects approximately superpose. When does it fail? When the adapters have learned conflicting representations: language A pushes certain attention heads into one subspace while domain B pushes the same heads into another. The combined model may regress on both tasks. Interference is more likely when adapters target the same modules at high rank.

A harder variant is the svd combination type: the delta matrices are summed, then re-factorised to rank r via SVD. This re-compresses the combined delta but loses the components below the rank cutoff. The cat type concatenates the B and A matrices, which doubles the rank (and may cause OOM at high ranks). The dare_linear and dare_ties methods add a pruning step first, dropping low-magnitude delta entries before merging, which reduces interference in practice (see the DARE paper, arxiv.org/abs/2311.03099).

Sequential stacking

Bottleneck adapters (Houlsby et al., 2019) insert small feed-forward modules inside transformer layers:

h -> LayerNorm -> down_proj (d -> r) -> GeLU -> up_proj (r -> d) -> h (residual)

Stacking a second adapter means inserting a second such module after the first, in the same layer. The residual stream flows through both. Because each adapter receives the output of the previous one, they are not independent: the second adapter effectively fine-tunes on top of a model that has already been shifted by the first.

AdapterFusion (Pfeiffer et al., 2020, arxiv.org/abs/2005.00247) generalises sequential composition into an explicit two-stage protocol. In stage one, each adapter is trained independently for its own task. In stage two, a small attention module is inserted after all the adapters; it learns to attend over their outputs and weight them per-layer, per-token. The gating parameters are the only new things trained in stage two. This keeps adapter knowledge non-destructive: no adapter is re-trained, and the attention head can in principle down-weight an irrelevant adapter to near zero.

The appeal is modularity. AdapterHub (Pfeiffer et al., arxiv.org/abs/2007.07779) built a public repository on top of this idea, letting practitioners download task-specific adapters and compose them at will. By 2021, the hub hosted hundreds of adapters for BERT and RoBERTa across dozens of languages and tasks.

Dynamic mixing: mixture-of-adapters

The logical endpoint of learned gating is treating adapters as experts in a sparse mixture. LoraHub (Huang et al., 2023, arxiv.org/abs/2307.13269) demonstrates a lightweight version of this for LoRA: given a small set of examples from a new task (no gradient, just forward passes), a gradient-free optimiser searches for per-adapter weights that minimise the few-shot loss. The adapter weights are fixed; only the scalar mixing coefficients are updated. The search converges in minutes and generalises to held-out examples.

X-LoRA (Buehler and Buehler, 2024, arxiv.org/abs/2402.07148) goes further by learning a deep gating mechanism that produces layer-wise and token-wise mixing coefficients. The gate sees the hidden state at each layer and outputs a soft routing distribution over the adapter pool. This is analogous to soft MoE applied to adapters rather than FFN experts. The cost is that inference now requires one forward pass per adapter pool entry to compute logits before routing, which is non-trivial at large pool sizes.

For production serving, the key insight from the Punica system (Chen et al., 2023, arxiv.org/abs/2310.18547) is that all of this composition can happen without duplicating base model weights: a single base model copy in GPU HBM serves a heterogeneous batch where each request applies a different adapter (or mixture of adapters), provided the kernel can handle per-request adapter offsets efficiently.

A concrete code sketch

The PEFT library makes linear interpolation straightforward:

from peft import PeftModel, LoraConfig

# Load a base model with two named adapters
model = PeftModel.from_pretrained(base, "adapter_spanish", adapter_name="spanish")
model.load_adapter("adapter_legal", adapter_name="legal")

# Blend them into a new adapter
model.add_weighted_adapter(
    adapters=["spanish", "legal"],
    weights=[0.5, 0.5],
    adapter_name="spanish_legal",
    combination_type="linear",
)
model.set_adapter("spanish_legal")

The call to add_weighted_adapter folds the blended delta into a new set of adapter tensors. No retraining, no new GPU passes. The resulting model has three named adapters; you can switch between them at zero cost.

For more surgical control, rank_pattern and alpha_pattern in LoraConfig let you assign different ranks to different layers, which matters when composing adapters of mismatched ranks: a direct linear sum of a rank-8 and rank-32 adapter is dominated by the higher-rank one unless weights are re-scaled.

When it falls down

Conflicting subspaces. Two adapters trained on orthogonal objectives (for example, a creative writing adapter and a code generation adapter) occupy different regions of weight space. Linear combination produces neither; it produces an averaged model that is mediocre at both. This is the same conflict seen in multi-task fine-tuning but compressed into a much smaller parameter budget.

Rank mismatch accumulation. When stacking three or more adapters sequentially, the effective perturbation to the weight matrix grows with depth. Even if each individual adapter is small relative to the base weight, their combined effect can destabilise learned representations - particularly in early transformer layers where representations are less specialised and small shifts propagate further.

Gating collapse. In AdapterFusion and similar soft-routing approaches, the attention gate can converge to always selecting one adapter, rendering the multi-adapter setup equivalent to a single adapter plus wasted compute. Regularising the gate (entropy penalties, dropout) helps but adds hyperparameter surface.

Catastrophic interference in non-LoRA adapters. Bottleneck adapters are not as cleanly decomposable as LoRA. Because they modify activations rather than weight matrices, their sequential composition depends on the order in which they were inserted and the residual connections around each. Swapping the order of two stacked adapters generally changes the result. LoRA's formulation as a weight delta is order-invariant for linear composition, which is one reason it has become the dominant paradigm for composition experiments.

Compute overhead at scale. LoraHub-style coefficient search runs on a small validation set via a gradient-free method (such as evolutionary search). For large pools - say 50 adapters - the search space grows combinatorially and the approach becomes expensive. Practical deployments tend to pre-select a small candidate set using task similarity metrics before running the coefficient search.

Further reading

• Pfeiffer et al. (2020), "AdapterFusion: Non-Destructive Task Composition for Transfer Learning": https://arxiv.org/abs/2005.00247
• Huang et al. (2023), "LoraHub: Efficient Cross-Task Generalisation via Dynamic LoRA Composition": https://arxiv.org/abs/2307.13269
• Buehler and Buehler (2024), "X-LoRA: Mixture of Low-Rank Adapter Experts": https://arxiv.org/abs/2402.07148
• HuggingFace PEFT add_weighted_adapter API reference: https://huggingface.co/docs/peft/main/en/package_reference/lora