LoRA Training Pitfalls

You spend three hours fine-tuning a 7B model with LoRA, eval loss curves look healthy, and then the model hallucinates on every domain-specific prompt you care about. The loss was fine. The checkpoint was wrong. This is not unusual. LoRA's elegance as a training method hides a handful of failure modes that are easy to miss precisely because they don't surface as obvious training crashes.

This concept unpacks the most consequential pitfalls: rank misconfiguration, the hidden learning-rate asymmetry between adapter matrices, wrong target modules, quantisation-induced initialisation errors, and catastrophic forgetting vs. under-adaptation. Knowing where each one hides lets you debug faster and design better fine-tuning runs.

How LoRA Works (the part that creates the pitfalls)

Recall the mechanics briefly, because each pitfall connects to a specific design choice. LoRA freezes the pre-trained weight matrix W_0 ∈ R^{d×k} and injects a low-rank bypass:

W = W_0 + BA,   where B ∈ R^{d×r},  A ∈ R^{r×k},  r << min(d, k)

A is initialised with Kaiming-uniform noise; B is initialised to zero. This ensures the adapter contributes nothing at the start of training (delta W = BA = 0), preserving the base model's behaviour on step zero. The final contribution is scaled by alpha / r before being added to the frozen weight.

Three variables control almost everything downstream: the rank r, the scaling ratio alpha / r, and which weight matrices receive adapters (target modules). Getting any one of them wrong produces a distinct class of problem.

Pitfall 1: Rank That is Too Low (or Too High)

The most common mistake is treating rank as a dial that only affects parameter count, when it actually bounds the expressivity of the adaptation.

Hu et al. (2021) found that for GPT-3 adaptation, rank 4 to 8 was sufficient for many NLP tasks because the task-specific signal lies in a low-dimensional subspace. But "many NLP tasks" is not "your task." Code generation, long-form reasoning, and domain shift from a narrow corpus routinely need higher rank. Biderman et al. (2024) showed that full fine-tuning implicitly learns perturbations whose effective rank is 10 to 100 times larger than typical LoRA configurations, and that LoRA substantially underperforms full fine-tuning on these harder adaptation targets.

The trap: loss curves converge at any rank because the model just learns what it can within the subspace you gave it. There is no "rank too low" warning in your training logs.

Practical guidance: start at r = 16 for general instruction following, r = 32 or higher for domain-heavy or code tasks. If your validation loss plateaus early compared to a full-finetune baseline, suspect rank before any other hyperparameter.

The opposite failure is also real. Setting rank very high (r = 128, r = 256) with the default scaling alpha / r causes the effective learning rate for the adapter to shrink proportionally. Kalajdzievski (2023) showed that this stunts learning with higher-rank adapters. The fix is use_rslora=True, which changes the scaling to alpha / sqrt(r), keeping the effective contribution stable as rank grows.

At high ranks, the default scaling shrinks the adapter's contribution so aggressively that you are effectively training with a much lower learning rate for the adapter while the frozen base stays fixed.

Pitfall 2: Learning Rate Asymmetry Between A and B

The original LoRA paper applies the same learning rate to both adapter matrices A and B. Hayou et al. (2024) proved that this is suboptimal. For large embedding dimensions, A and B need to evolve at different rates for the product BA to converge efficiently.

Intuitively: A projects from high-dimensional input (dimension k, e.g., 4096) down to rank r. B projects back up. They sit at different scales in the loss landscape, and a single learning rate cannot be optimal for both simultaneously.

The fix is LoRA+ (also from Hayou et al.), which sets a ratio between the learning rates of A and B (typically lr_B = 16 * lr_A). The Hugging Face PEFT library exposes this via loraplus_lr_ratio. In practice this produces 1-2% gains on standard benchmarks and can deliver up to 2x speedup in convergence, at essentially no cost.

If you are using a shared learning rate scheduler across your adapter and base model (even if base is frozen, there may be layer-norm or embedding parameters that are unfrozen), verify the adapter learning rate is not being swept too low by a cosine schedule late in training. Adapters often need a higher base LR than the rest of the model.

Pitfall 3: Wrong Target Modules

By default, many training scripts apply LoRA only to q_proj and v_proj (query and value projections in attention). This is fine for the tasks Hu et al. tested. It is often wrong for everything else.

Practical consequences of under-targeting: - Skipping k_proj means the attention score matrix is partially adapted; the model can shift what it attends to (V) but not how it computes relevance (QK). - Skipping MLP layers (e.g., gate_proj, up_proj, down_proj in LLaMA-family models) means all feed-forward knowledge is frozen. For domain adaptation with new vocabulary or concepts, this is usually too restrictive. - Skipping the lm_head or embedding layer matters when you've added new tokens. Without modules_to_save covering these, new tokens never get trained.

The HuggingFace PEFT library lets you inspect which modules were targeted with model.print_trainable_parameters(). Check this before every training run; it is easy to misconfigure target_modules with a string pattern that silently matches nothing.

A reasonable starting point for LLaMA-family models:

target_modules = [
    "q_proj", "k_proj", "v_proj", "o_proj",
    "gate_proj", "up_proj", "down_proj",
]

Adding all seven attention and MLP projections increases parameter count modestly (roughly 2-3x versus q/v only at the same rank) but covers the full information flow.

Pitfall 4: Quantisation and Initialisation Errors (QLoRA)

QLoRA loads the base model in 4-bit NF4 quantisation before adding LoRA adapters. This introduces a subtle but important initialisation hazard. The standard Kaiming-uniform initialisation for A was designed assuming the base weights are in full precision. When the base is quantised, the initial residual between quantised weights and their true values is non-trivial; if the adapter is initialised independently, it starts from a position that ignores the quantisation error.

LoftQ (Liu et al., 2023) addresses this by jointly initialising the quantised backbone and LoRA adapters so the sum approximates the original full-precision weights as closely as possible. The HuggingFace PEFT docs explicitly warn: when using QLoRA, initialise with init_lora_weights="loftq" via LoftQConfig and do NOT quantise the model before applying LoftQ initialisation (the order matters).

The symptom of skipping this: the model trains normally but exhibits higher-than-expected perplexity or poor out-of-domain generalisation, because the adapter is compensating for quantisation error in addition to learning the task.

Pitfall 5: The Forgetting vs. Under-Adaptation Trade-off

Biderman et al. (2024) identified a property that sounds like a benefit but is a trap in disguise: LoRA forgets less than full fine-tuning. On tasks outside the training domain, LoRA preserves the base model's original performance better. This is useful when you care about the model's general capability.

But when your fine-tuning task genuinely requires overwriting old representations (e.g., instilling a very different writing style, shifting factual priors, teaching a new coding idiom), LoRA's reluctance to depart from the base can cause under-adaptation. The model seems to "resist" the new data. Increasing rank helps, but so does unfreezing certain layers entirely via modules_to_save rather than adapting them with low-rank projections.

The practical tension is real: aggressive LoRA (high rank, all modules) starts to approach full fine-tuning in parameter count while losing the inference efficiency advantage. Choosing the right point on that curve requires knowing whether your task needs departure from the base or refinement within it.

When It Falls Down

• Very short training data (< 1k examples): Low rank is usually better here, but the adapter can overfit even within a low-rank subspace if the dataset is tiny. Regularise with dropout on LoRA weights (lora_dropout=0.05) and a cosine LR schedule.
• Multi-task fine-tuning with LoRA: A single set of adapters cannot represent multiple divergent task distributions well. The solution is task-specific adapter sets (LoRA-mix or adapter routing), not a single high-rank adapter trained on a mixed batch.
• Continual learning over successive LoRA checkpoints: Merging multiple LoRA adapters trained on different tasks via weight arithmetic (A + B + C) produces interference. The merged model is not a sum of the fine-tuned capabilities; off-diagonal cross-task interference degrades all of them.
• Inference with merged weights: Merging LoRA back into the base model (merge_and_unload()) is lossless in fp16/bf16 but introduces small numerical errors in quantised models. Do not merge-and-quantise; keep the quantised base and load adapters separately.

Further Reading

• Hu et al. (2021), "LoRA: Low-Rank Adaptation of Large Language Models" - the foundational paper with the rank-deficiency analysis: https://arxiv.org/abs/2106.09685
• Biderman et al. (2024), "LoRA Learns Less and Forgets Less" - empirical comparison against full fine-tuning, rank implications: https://arxiv.org/abs/2405.09673
• Hayou et al. (2024), "LoRA+: Efficient Low Rank Adaptation of Large Models" - learning rate asymmetry fix: https://arxiv.org/abs/2402.12354
• Kalajdzievski (2023), "A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA" (rsLoRA) - scaling at high rank: https://arxiv.org/abs/2312.03732