LoRA Is Not Cheap Full Fine-Tuning: What Low-Rank Adaptation Actually Changes

The standard mental model of LoRA is a compression story. Full fine-tuning updates every weight; LoRA updates a low-rank stand-in for the same update; the result is approximately the same model at a fraction of the memory. Under that model the only interesting question is how much rank you need before the approximation stops hurting.

Two independent findings from 2024 say the compression story is wrong in a way that matters.

The first: on programming and mathematics, in both instruction tuning and continued pretraining regimes, LoRA substantially underperforms full fine-tuning at standard ranks, while better preserving the base model's capabilities outside the target domain. The same paper measured full fine-tuning learning weight perturbations with rank 10 to 100 times greater than typical LoRA configurations (Biderman et al., 2024, LoRA Learns Less and Forgets Less, TMLR, arXiv:2405.09673).

The second: LoRA and full fine-tuning produce weight matrices whose singular value decompositions have visibly different structure. LoRA introduces high-ranking singular vectors that full fine-tuning does not produce, which the authors named intruder dimensions, and showed that forgetting concentrates in them (Shuttleworth et al., 2024, LoRA vs Full Fine-tuning: An Illusion of Equivalence, arXiv:2410.21228).

Put together: matching downstream accuracy does not mean matching the solution. LoRA and full fine-tuning land in different places, with different generalisation and different forgetting behaviour, and the difference is measurable in the weights themselves.

Why this matters: If LoRA were an approximation, the design question would be "how much quality am I giving up?" It is not an approximation, so the question is "which of these two different behaviours do I want?" For a model that must stay generalist while acquiring a skill, less learning and less forgetting is the feature, not the compromise. For a model that must absorb a genuinely new domain, it is a trap.

TL;DR

• LoRA freezes \(W_0\) and trains a rank-\(r\) update \(BA\), cutting trainable parameters by up to 10,000x and optimiser memory by roughly 3x versus full fine-tuning on GPT-3 175B, with no added inference latency once merged (Hu et al., 2021, arXiv:2106.09685).
• The memory saving is almost entirely optimiser state and gradients, not weights. The frozen base still has to be resident, which is why QLoRA quantises it.
• LoRA learns less than full fine-tuning on hard domain shifts and forgets less outside the target domain, and it beats weight decay and dropout as a forgetting mitigation.
• Full fine-tuning finds updates of far higher effective rank than LoRA configurations use, which is one mechanistic explanation for the learning gap.
• LoRA's solutions contain intruder dimensions absent from full fine-tuning, and damping those singular values restores pre-training knowledge with little downstream loss.
• QLoRA made 65B fine-tuning fit on one 48GB GPU while preserving 16-bit fine-tuning task performance, via NF4, double quantisation, and paged optimisers (Dettmers et al., 2023, arXiv:2305.14314).
• The strongest practical argument for LoRA is not training cost. It is serving: thousands of adapters over one base model, with S-LoRA reporting up to 4x throughput over prior systems and orders of magnitude more adapters served (Sheng et al., 2023, arXiv:2311.03285).
• Rank is the wrong first knob. Which modules you adapt matters more, and learning rate matters more than both.

At a Glance

flowchart LR
    X["Input x"] --> W["Frozen W0<br/>d x k"]
    X --> A["A: r x k<br/>trainable"]
    A --> B["B: d x r<br/>trainable"]
    W --> S["Sum"]
    B -->|"scaled by alpha/r"| S
    S --> Y["Output h"]
    M["Merge at deploy:<br/>W = W0 + BA"] -.-> S

    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff

    class X blue
    class W slate
    class A,B purple
    class S,Y emerald
    class M slate

The trainable path is the two purple matrices. Everything else is frozen, and at deployment the whole detour collapses back into a single weight matrix.

Before Low-Rank Adaptation

Adapting a large pre-trained model to a task without retraining all of it is an old goal, and LoRA is the third or fourth serious attempt at it.

timeline
    title The road to parameter-efficient fine-tuning
    2019 : Houlsby adapters
         : Small bottleneck MLPs inserted between layers, trained while the base stays frozen
    2020 : Intrinsic dimensionality (Aghajanyan et al.)
         : Fine-tuning objectives have surprisingly low intrinsic dimension, motivating low-rank updates
    2021 : Prefix tuning and prompt tuning
         : Adapt by learning continuous vectors in the input or key-value space, no weight change
    2021 : LoRA (Hu et al.)
         : Low-rank update merged into the weights, so inference cost is unchanged
    2023 : QLoRA (Dettmers et al.)
         : 4-bit frozen base plus LoRA, 65B fine-tuning on a single 48GB GPU
    2023 : S-LoRA and multi-adapter serving
         : Thousands of adapters batched against one base model in production
    2024 : DoRA, and the equivalence question
         : Magnitude/direction decomposition; evidence that LoRA and full FT are structurally different

Houlsby-style adapters worked but added modules to the forward pass, so inference got slower, which mattered enormously for serving (Houlsby et al., 2019, arXiv:1902.00751). Prefix and prompt tuning avoided touching weights entirely by learning continuous vectors, but consumed sequence budget and were finicky to optimise (Li & Liang, 2021, arXiv:2101.00190; Lester et al., 2021, arXiv:2104.08691).

The conceptual ingredient came from a different direction. Aghajanyan and colleagues showed that fine-tuning objectives have low intrinsic dimension: you can reparameterise into a few thousand dimensions and still reach good task performance (Aghajanyan et al., 2020, arXiv:2012.13255). If the useful update lives in a low-dimensional subspace, then parameterising it as low-rank is not a compromise but a reasonable prior.

LoRA's contribution was to put that prior in a form with no inference penalty. Because the update is linear and additive, \(BA\) can be folded into \(W_0\) after training, so a deployed LoRA model is architecturally identical to the base.

How LoRA Actually Works

The parameterisation

For a pre-trained weight matrix \(W_0 \in \mathbb{R}^{d \times k}\), LoRA freezes \(W_0\) and represents the update as a product of two much thinner matrices:

with \(B \in \mathbb{R}^{d \times r}\), \(A \in \mathbb{R}^{r \times k}\), and \(r \ll \min(d, k)\). \(A\) is initialised from a random Gaussian and \(B\) to zero, so \(\Delta W = 0\) at step zero and training begins exactly at the pre-trained model. The scaling factor \(\alpha / r\) exists so that changing \(r\) does not require retuning the learning rate.

Trainable parameters drop from \(dk\) to \(r(d + k)\). For \(d = k = 4096\) and \(r = 16\), that is 16.7M down to 131k, a factor of 128 for that matrix.

Where the memory actually goes

The commonly quoted "LoRA saves memory" needs decomposition, because the saving is not where people assume. Training memory has four parts:

Figures are order-of-magnitude estimates for a 7B model at 2 bytes per parameter for weights and 8 bytes per trainable parameter for Adam moments in FP32.

The optimiser state is the dominant term and it scales with trainable parameters, which is why LoRA's saving is so large. But the base weights do not shrink at all, and that residual is what QLoRA attacks: quantise the frozen base to 4-bit NormalFloat, keep LoRA adapters in 16-bit, and add double quantisation plus paged optimisers to handle the remaining spikes. The reported result is a 65B model fine-tuned on a single 48GB GPU while preserving full 16-bit fine-tuning task performance, with the Guanaco family reaching 99.3% of ChatGPT's level on the Vicuna benchmark after 24 hours of single-GPU training (Dettmers et al., 2023, arXiv:2305.14314).

[IMAGE: Stacked bar chart of training memory for a 7B model, comparing full fine-tuning and LoRA r=16, with four segments per bar: base weights, gradients, optimiser state, activations. The base-weight segment is identical in both; the optimiser segment dominates the full fine-tuning bar and nearly vanishes in the LoRA bar. Overlay a bracket on the base-weight segment labelled "what QLoRA attacks". Caption: "LoRA removes the two segments that scale with trainable parameters. Quantisation is what removes the third."]

Rank is not the knob you think

The instinct is to tune \(r\) first. Practice and published ablations point elsewhere.

Which modules you adapt matters more than how much rank you give them. LoRA's original experiments adapted attention projections only. Later practice, and the QLoRA ablations in particular, found that adapting all linear layers, including the MLP projections, matters more than raising rank on a subset. A rank-8 adapter on every linear layer usually beats a rank-64 adapter on \(W_q\) and \(W_v\) alone.

Learning rate matters more than either. LoRA typically wants a learning rate an order of magnitude higher than full fine-tuning, because the effective update passes through the \(\alpha/r\) scaling and the zero-initialised \(B\). A LoRA run that underperforms is more often under-tuned than under-ranked.

\(\alpha\) is not independent of \(r\). Setting \(\alpha = 2r\) is a common default and keeps the effective scale stable as rank changes. Treating \(\alpha\) and \(r\) as two free knobs invites confounded sweeps.

[IMAGE: Line chart of downstream task accuracy against LoRA rank (4, 8, 16, 32, 64, 128) with three curves: attention-projections-only adaptation, all-linear-layers adaptation, and a horizontal dashed line for full fine-tuning. The all-linear curve at rank 8 sits above the attention-only curve at rank 64. Caption: "Module coverage moves the curve further than rank does. Most LoRA sweeps tune the wrong axis first."]

Seeing It in Motion

Training and deployment paths

sequenceDiagram
    participant D as Task data
    participant F as Frozen base W0
    participant L as LoRA A, B
    participant O as Optimiser
    participant S as Serving

    D->>F: forward pass
    F->>L: activations
    L->>L: compute alpha/r * BAx
    L-->>F: added to W0x
    F-->>O: loss, backward
    O->>L: update A and B only
    Note over F: no gradient, no optimiser state
    L->>S: ship adapter (tens of MB)
    Note over S: merge for one tenant,<br/>or keep separate for many

The two ways to deploy an adapter

flowchart TB
    subgraph Merge["Merged deployment"]
        direction TB
        M1["W = W0 + BA"] --> M2["One model artifact"]
        M2 --> M3["Zero inference overhead"]
        M3 --> M4["One adapter per replica"]
    end
    subgraph Multi["Multi-adapter serving"]
        direction TB
        S1["Base weights resident once"] --> S2["Adapter A, B kept separate"]
        S2 --> S3["Batched heterogeneous requests"]
        S3 --> S4["Thousands of adapters,<br/>small per-request overhead"]
    end

    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef teal fill:#0e7490,stroke:#22d3ee,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff

    class M1,M2,M3 emerald
    class M4 amber
    class S1,S2,S3,S4 teal

Merging gives literally zero inference overhead and costs you the ability to serve more than one variant per replica. Keeping adapters separate costs a small per-token overhead and turns a fleet of 200 fine-tunes from impossible into routine.

Why forgetting concentrates where it does

stateDiagram-v2
    [*] --> Pretrained: base weights, spectrum shaped by pre-training
    Pretrained --> FullFT: full fine-tuning
    Pretrained --> LoRAFT: low-rank update
    FullFT --> HighRank: high-rank perturbation, spectrum stays aligned
    LoRAFT --> Intruder: new high-ranking singular vectors appear
    Intruder --> Forgetting: pre-training knowledge degrades along those directions
    Intruder --> Damped: damp intruder singular values
    Damped --> Recovered: pre-training performance largely restored

Watch It Run

The companion diagram (lora-is-not-cheap-full-fine-tuning.drawio) animates one training step through a LoRA-adapted layer: activations flowing through the frozen base and the low-rank branch in parallel, the gradient loop that touches only \(A\) and \(B\), and the merge path that collapses the branch at deployment.

By the Numbers

Sources: LoRA (Hu et al., 2021, arXiv:2106.09685), QLoRA (Dettmers et al., 2023, arXiv:2305.14314), DoRA (Liu et al., ICML 2024, arXiv:2402.09353), adapters (Houlsby et al., 2019, arXiv:1902.00751). Each figure is the authors' own reported result on their own setup; the trainable-parameter ranges are typical configurations rather than measurements.

The quality comparison that matters most is not in this table because it does not reduce to one number. Biderman and colleagues compared LoRA against full fine-tuning on code and mathematics, across instruction tuning (roughly 100K prompt-response pairs) and continued pretraining (20B unstructured tokens), and found LoRA substantially behind in the target domain at standard ranks while holding onto out-of-domain performance that full fine-tuning lost. They also measured full fine-tuning producing weight perturbations of rank 10 to 100 times higher than typical LoRA configurations, which is the cleanest available mechanistic account of the gap.

[IMAGE: Two-panel plot from the learning/forgetting tradeoff. Left panel: target-domain accuracy on code and math, with full fine-tuning above LoRA at every rank tested but the gap narrowing at high rank. Right panel: out-of-domain benchmark retention, with LoRA above full fine-tuning throughout. Caption: "The same experiment produces opposite orderings depending on which axis you measure. There is no configuration that wins both."]

A Concrete Example

A team adapts a 7B base model to write internal SQL against a proprietary schema. They have 12,000 curated query pairs.

Setup. \(d = k = 4096\) for the attention projections, 32 layers, adapters on all linear layers including MLP. Rank 16, \(\alpha = 32\).

Parameter count. Per attention projection, \(r(d+k) = 16 \times 8192 = 131{,}072\) parameters against \(4096^2 = 16.8\)M for the full matrix. Across four attention projections and three MLP projections per layer, with MLP matrices of shape \(4096 \times 11008\), the adapters total roughly 40M trainable parameters against 7B, about 0.57%.

Memory. Base weights in BF16: about 14 GB. Adam moments over 40M trainable parameters in FP32: about 320 MB, against roughly 56 GB for full fine-tuning. The run fits on one 40GB GPU with room for activations; the full fine-tune would need four.

First result. Exact-match accuracy on held-out queries reaches 71%, against 78% for a full fine-tune the team ran once as a reference. The initial reaction is to raise rank to 64. That moves accuracy to 73%, at four times the adapter size.

What actually helped. Two changes, neither of them rank. The learning rate goes from 1e-4 to 5e-4, worth about four points on its own. Then they check module coverage and find the MLP down_proj was excluded by a config typo; restoring it adds another two. Final: 78.5%, at rank 16, matching the full fine-tune with 0.57% of the trainable parameters.

The part that only shows up later. The full fine-tune, evaluated on the team's general coding eval, dropped from 46% to 31%. The LoRA model held at 44%. The assistant is expected to answer general Python questions as well as write SQL, so the LoRA model is the better product even where the full fine-tune had matched it on the target task. That result is what "learns less, forgets less" looks like in a deployment.

[IMAGE: Grouped bar chart with two metric groups, target-task SQL exact match and general coding eval retention, and three bars in each: base model, LoRA, full fine-tune. Full fine-tune wins the first group narrowly and loses the second heavily. Caption: "Evaluating only on the target task would have selected the model that broke the product."]

Where It Breaks

Genuinely new domains

The failure mode with the clearest evidence is continued pretraining on a domain the base model has little exposure to. A low-rank update cannot express a high-rank change, and when the required perturbation genuinely is high rank, no learning-rate tuning recovers it. If your adaptation is closer to teaching a new language than to shaping a style, LoRA at standard ranks will underperform and raising rank until it does not is just full fine-tuning with extra steps.

[IMAGE: Singular value spectra of the same weight matrix after full fine-tuning and after LoRA, plotted as sorted singular values on a log axis with the pre-trained spectrum as a grey reference line. The full fine-tuning curve tracks the reference closely; the LoRA curve shows a small number of high-ranking spikes departing from it, annotated "intruder dimensions". Caption: "The structural difference is visible in the spectrum, not in the benchmark score."]

Intruder dimensions and sequential adaptation

The equivalence paper's finding has a specific operational consequence. LoRA models accumulate intruder dimensions across successive adaptations, and this degrades continual learning performance relative to full fine-tuning. A pipeline that applies adapter after adapter to the same lineage is exactly the case where the structural difference compounds, and the failure looks like unexplained quality drift rather than an error.

The merge is not always safe

Merging \(BA\) into \(W_0\) is exact in 16-bit. Merging into a quantised base is not: the merged weights must be re-quantised, and the quantisation error can undo part of what was learned. QLoRA-trained adapters are usually served against a quantised base unmerged for this reason, which reintroduces a small inference overhead.

Multi-adapter serving has a real cost

Serving many adapters against one base means the per-request path can no longer be a single fused GEMM. S-LoRA's contribution was unified paging for adapter weights and custom CUDA kernels for heterogeneous batching, reporting up to 4x throughput improvement over HuggingFace PEFT and vLLM while serving thousands of adapters. The overhead is small but nonzero, and adapter rank heterogeneity across tenants makes batching harder.

Rank does not extrapolate across tasks

An \(r\) that works for style adaptation is not evidence about an \(r\) for reasoning. The intrinsic dimension of the task determines it, and nothing about the model or the dataset size predicts it in advance. Sweep it per task family, not per model.

Alternative Designs

DoRA deserves the closest look for teams whose LoRA runs underperform. It decomposes pre-trained weights into magnitude and direction components and applies LoRA only to the directional part, which its authors report consistently outperforms LoRA on LLaMA, LLaVA and VL-BART across commonsense reasoning, visual instruction tuning, and image and video-text understanding, with no additional inference overhead.

How It Is Used in Practice

The dominant production pattern is not one team fine-tuning one model. It is a platform serving many variants: per-customer tone, per-vertical vocabulary, per-team formatting, all against one base. That pattern is only economical because adapters are tens of megabytes and can share a resident base model, and it is the strongest argument for LoRA that has nothing to do with training cost.

Concretely, this turns the cold-start problem from impossible into routine. A fleet serving 200 full fine-tunes needs 200 warm replicas. A fleet serving 200 adapters over one warm base swaps a few hundred megabytes per request class.

The operational details that decide whether it works: adapter versioning must be tied to base-model versioning, because an adapter trained against one base is not valid against another and nothing checks this at load time. Rank should be standardised across a fleet where possible, since heterogeneous ranks complicate batched kernels. And the eval suite must include out-of-domain retention, or the pipeline will happily select adapters that solve the target task by degrading everything else.

For training, the pragmatic default that has emerged: adapt all linear layers, start at rank 16 with \(\alpha = 32\), use a learning rate several times higher than you would for full fine-tuning, and only then consider rank. If that plateaus below requirement, try DoRA before trying rank 128.

[IMAGE: Serving architecture diagram showing one base model resident in GPU memory with a paged adapter pool beneath it, incoming requests tagged by tenant routed to their adapter, and a batched kernel processing requests with different adapters in the same batch. Caption: "Multi-adapter serving is the economic case for LoRA. Training cost is the smaller argument."]

Insights Worth Remembering

1. LoRA is not an approximation of full fine-tuning; it is a different optimisation. The two produce weight matrices with structurally different spectra, so matching downstream accuracy does not imply matching behaviour anywhere else.

2. Less learning and less forgetting are the same property viewed from two sides. A constrained update cannot move far in the target domain and cannot move far away from the base either. Which side you call a benefit depends on what the model has to keep doing.

3. The memory saving is optimiser state, not weights. Gradients and Adam moments scale with trainable parameters; the frozen base does not shrink. Understanding this is what makes QLoRA's design obvious rather than clever.

4. Module coverage beats rank. Adapting every linear layer at low rank generally outperforms adapting a subset at high rank, and the most common LoRA misconfiguration is silently omitting the MLP projections.

5. A LoRA run that underperforms is usually under-tuned, not under-ranked. Learning rate is the first thing to move, typically to several times the full fine-tuning value.

6. Zero inference overhead is a property of merging, and merging is not always available. Against a quantised base, or in a multi-adapter deployment, you keep the branch and pay a small cost.

7. Intruder dimensions give catastrophic forgetting a location. Forgetting concentrating in identifiable singular directions, and damping them restoring pre-training knowledge, turns a diffuse phenomenon into something measurable and partly reversible.

8. Sequential adaptation is where LoRA's structural difference compounds. Stacking adapters over the same lineage accumulates intruder dimensions, so continual-learning pipelines are the case to be most careful about.

9. The real argument for LoRA is serving economics. Thousands of adapters over one resident base is a capability full fine-tuning simply does not have, at any budget.

10. Rank is a property of the task, not of the model. Nothing about parameter count or dataset size predicts the intrinsic dimension of the adaptation you are attempting.

Open Questions

How much of the learning gap is rank, and how much is optimisation? The measured rank difference between full fine-tuning and LoRA updates is a strong hypothesis, but higher-rank LoRA does not close the gap as cleanly as a pure capacity story predicts. Whether the residual is an optimisation-landscape effect or an initialisation effect is not settled.

Can intruder dimensions be prevented rather than damped? Post-hoc damping restoring pre-training knowledge is evidence that the harmful component is separable. Whether a regulariser or initialisation scheme can avoid producing it in the first place, without giving up the learning that LoRA does achieve, is open.

Is DoRA's advantage general or architectural? Its reported gains are consistent across the tasks its authors tested. Independent replication across more base models and more domains is thinner than the adoption rate would suggest.

What is the right unit for multi-tenant adaptation? Adapters, prompts, retrieval, and routing to specialised models all solve "one base, many behaviours". The economics differ by workload and nobody has published a clean decision procedure.

Does the picture survive at frontier scale? Nearly all published LoRA-versus-full comparisons run at 7B to 70B. Whether the learning gap narrows, widens, or changes character at frontier scale is unmeasured in public, largely because the comparison requires a full fine-tune of a frontier model.

Sources and Further Reading

Foundational papers

1. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019). "Parameter-Efficient Transfer Learning for NLP." ICML 2019. arXiv:1902.00751

2. Aghajanyan, A., Zettlemoyer, L., & Gupta, S. (2020). "Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning." arXiv:2012.13255

3. Li, X. L., & Liang, P. (2021). "Prefix-Tuning: Optimizing Continuous Prompts for Generation." ACL 2021. arXiv:2101.00190

4. Lester, B., Al-Rfou, R., & Constant, N. (2021). "The Power of Scale for Parameter-Efficient Prompt Tuning." EMNLP 2021. arXiv:2104.08691

5. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). "LoRA: Low-Rank Adaptation of Large Language Models." arXiv:2106.09685

Efficiency and variants

6. Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). "QLoRA: Efficient Finetuning of Quantized LLMs." NeurIPS 2023. arXiv:2305.14314

7. Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F., Cheng, K.-T., & Chen, M.-H. (2024). "DoRA: Weight-Decomposed Low-Rank Adaptation." ICML 2024. arXiv:2402.09353

8. Sheng, Y., Cao, S., Li, D., Hooper, C., Lee, N., Yang, S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., Gonzalez, J. E., & Stoica, I. (2023). "S-LoRA: Serving Thousands of Concurrent LoRA Adapters." arXiv:2311.03285

The equivalence question

9. Biderman, D., Portes, J., Ortiz, J. J. G., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S., Chiley, V., Frankle, J., Blakeney, C., & Cunningham, J. P. (2024). "LoRA Learns Less and Forgets Less." TMLR. arXiv:2405.09673

10. Shuttleworth, R., Andreas, J., Torralba, A., & Sharma, P. (2024). "LoRA vs Full Fine-tuning: An Illusion of Equivalence." arXiv:2410.21228

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.