DoRA: Weight-Decomposed LoRA

Standard LoRA with rank 16 on LLaMA-7B reaches 74.7% average accuracy across eight commonsense reasoning benchmarks. Full fine-tuning reaches roughly 79%. DoRA, with the same rank and the same parameter budget, reaches 78.4% - recovering most of that gap by changing not what parameters are trained, but how the weight update is structured.

That five-point gap is not a minor tuning artefact. It traces to a fundamental constraint in LoRA's update geometry: magnitude and direction are forced to move together, whereas full fine-tuning moves them independently. DoRA was designed to remove that constraint.

What LoRA Gets Wrong About Weight Updates

LoRA freezes the pre-trained weight matrix W and adds a low-rank perturbation:

W' = W + BA      (B ∈ ℝ^{d×r}, A ∈ ℝ^{r×k}, r ≪ min(d,k))

The key empirical observation in the DoRA paper (Liu et al., ICML 2024) is that, when you measure the Pearson correlation between magnitude changes and directional changes across weight matrices during LoRA training, you get a correlation of +0.83. Magnitude and direction are coupled: when one grows, so does the other, almost deterministically.

Full fine-tuning shows a correlation of -0.62 - a weak negative relationship, meaning the two dimensions adjust largely independently and sometimes in opposite directions. Pre-trained weights already encode useful structure; effective adaptation often calls for refining direction while holding magnitude roughly steady, or vice versa.

LoRA cannot do this. Its rank decomposition modifies a single additive delta, so any factorisation of that delta into "magnitude change" and "direction change" is entangled. DoRA addresses this at the decomposition level.

The Magnitude-Direction Decomposition

Any matrix W can be written as:

W = m · (V / ‖V‖_c)

where:

• m ∈ ℝ^{1×k} is the magnitude vector (one scalar per output column, capturing the column-wise L2 norms of W).
• V ∈ ℝ^{d×k} is the directional matrix (W rescaled so each column has unit norm).
• ‖·‖_c denotes column-wise vector norms.

This decomposition is not an approximation - it is exact for any matrix. Its value is that it makes the two axes of variation explicit and separately addressable.

DoRA initialises from the pre-trained weight W₀ by computing m₀ = ‖W₀‖_c and V₀ = W₀ column-normalised. It then:

1. Makes m a learnable parameter (a single vector of size k, added overhead is negligible).
2. Applies LoRA to the directional component, not to W directly.

The fine-tuned weight becomes:

W' = m̄ · (W₀ + BA) / ‖W₀ + BA‖_c

where m̄, B, and A are all trainable; W₀ is frozen. The magnitude vector scales the final result; the LoRA matrices B and A steer direction.

At initialisation, B = 0 (standard LoRA practice), so W' = m̄ · W₀/‖W₀‖_c = m₀ · W₀/‖W₀‖_c = W₀. No-op at start, just as in LoRA.

Why This Matters for Learning Dynamics

The decoupling has a concrete training consequence. During gradient descent, m can grow or shrink a column's overall scale without any constraint on what B and A are doing. The gradient flowing into m is orthogonal (in effect) to the gradient flowing into BA.

This mirrors how full fine-tuning behaves, which is precisely why DoRA's correlation between magnitude and directional changes (-0.31) is much closer to full fine-tuning (-0.62) than to LoRA (+0.83).

The practical payoff is most visible at low ranks, where LoRA's expressive capacity is already limited. When rank is constrained, every parameter must count; entangling magnitude and direction wastes representational capacity. DoRA separates the concerns.

A comparison across LLaMA variants on eight commonsense tasks shows the gains are consistent and scale with model capacity:

The gains on LLaMA3-8B (+4.4 pp) are striking given that LLaMA3 is already a stronger base.

Training and Inference Overhead

Parameter overhead. Compared to LoRA, DoRA adds one magnitude vector m of size k per adapted layer. For a 7B model targeting all projection matrices (roughly 32 layers × 4 projections × hidden dim ~4096), the extra parameters amount to around 500k scalars - under 0.01% of total model parameters. This is genuinely negligible.

Training overhead. The column normalisation in the forward pass adds a norm computation per layer per step. In practice this adds roughly 20-40% wall-clock time per training step compared to LoRA, depending on batch size and hardware. The HuggingFace PEFT documentation notes explicitly that DoRA introduces a bigger overhead than pure LoRA during training.

Inference overhead. Zero. Before deployment you merge the adapted weight back into a single matrix:

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    use_dora=True,          # the only change from LoRA
)
model = get_peft_model(base_model, config)

# After training, merge for zero-overhead inference:
model.merge_adapter()

The use_dora=True flag in HuggingFace PEFT is the complete API surface. The merged weight is a plain nn.Linear with no extra runtime.

When It Falls Down

Training memory. The normalisation operation requires materialising intermediate tensors proportional to the layer's output dimension. At bf16 on a 70B model training at batch size 4, this can add 2-4 GB of peak activation memory over LoRA. For memory-constrained QLoRA setups, this headroom may not exist.

Very high ranks. At rank 64+ the directional component already has enough capacity to approximate full fine-tuning on its own. The magnitude decoupling provides diminishing returns, and the additional compute during training becomes harder to justify.

Convolutional layers. The PEFT implementation supports nn.Linear and Conv2D, but the column-wise norm semantics for 4D weight tensors in conv layers are less obviously correct; results on vision models are less well-validated than on transformer LLMs.

Quantised training (QDoRA). When the base weights are 4-bit quantised (NF4 via bitsandbytes), the column normalisation operates on dequantised values. The quantisation error introduces noise into the magnitude vector's gradient that does not exist in standard QLoRA. Whether this harms convergence depends on the task and rank; there is no clear guidance in the original paper.

Batch size sensitivity. The column-norm computation is data-independent (it operates on weights, not activations), so there is no batch size sensitivity in the DoRA update itself. However, the magnitude parameters m can overfit on small datasets because they have a direct, unconstrained path to scale any column arbitrarily. Applying a small weight decay (1e-4 to 1e-3) specifically to m is advisable for tasks with fewer than ~10k training examples.

Further Reading

• Liu, S.-Y. et al. (2024). "DoRA: Weight-Decomposed Low-Rank Adaptation." ICML 2024 (Oral). https://arxiv.org/abs/2402.09353
• Hu, E. et al. (2021). "LoRA: Low-Rank Adaptation of Large Language Models." https://arxiv.org/abs/2106.09685
• HuggingFace PEFT LoraConfig API reference (see use_dora parameter). https://huggingface.co/docs/peft/package_reference/lora