Parameter-Efficient Fine-Tuning
LoRA, QLoRA, DoRA, adapters and soft prompts — adapting big models on small budgets.
20concepts
140flashcards
145minutes of reading
- 01 Composing and Stacking Adapters Multiple trained adapters can be combined sequentially, by weighted sum, or through attention-based gating to build new capabilities without retraining the base model.
- 02 DoRA: Weight-Decomposed LoRA DoRA decomposes pre-trained weights into magnitude and direction components, then applies LoRA exclusively to the directional part, closing most of the accuracy gap between LoRA and full fine-tuning without adding inference overhead.
- 03 LoRA for Long-Context Adaptation Extending a model's context window via LoRA requires coordinating low-rank weight updates with position-encoding rescaling, and ignoring either side reliably degrades performance on long sequences.
- 04 NF4 and Double Quantisation NF4 is a 4-bit data type matched to the normal distribution of pretrained weights, and double quantisation further compresses the quantisation constants themselves, together enabling 65B-parameter models to fine-tune on a single 48 GB GPU via QLoRA.
- 05 Serving Many LoRA Adapters How specialised inference systems batch requests across hundreds of distinct LoRA adapters without duplicating the base model weights on GPU.