NOTES / FINE-TUNING AND ADAPTATION

LoRA and QLoRA

Parameter-efficient fine-tuning: train small low-rank adapters on a frozen model, and in QLoRA keep the frozen model in 4 bits. What it saves, and what it costs.

UPDATED 18 NOV 2026#fine-tuning#lora#qlora#peft

The problem

Full fine-tuning updates every weight, so you must store gradients and optimiser state for all of them, plus a full copy of the model for each task. Hu and colleagues note that the major downside is storing as many parameters as the original model1.

The idea

LoRA freezes the pretrained weights and adds a small trainable update to selected weight matrices. The update is the product of two thin matrices, B and A, whose inner dimension r (the rank) is small1. The hypothesis is that the change needed to adapt a model has low intrinsic rank, so a very small r can be enough; for GPT-3 175B they report that r of one or two sufficed on some tasks even though the full dimension is 12,2881.

Diagram: input goes through the frozen weight W and through the trainable pair A then B; the two outputs are added

Diagram: my own.

Reported effects for GPT-3 175B compared with fine-tuning with Adam: about 10,000 times fewer trainable parameters and about three times less GPU memory2. Because the update is a linear addition, it can be merged into W after training, which is why the authors report no extra inference latency1. You can also keep adapters separate and swap them per task or per customer, storing a few megabytes instead of a whole model (my summary of the design).

Practical settings, which are conventions rather than rules: a rank somewhere in the range of roughly 4 to 32 is common5, applied to the attention projections and often the feed-forward layers, with a scaling factor and a learning rate that you tune.

QLoRA

QLoRA keeps the frozen base model in 4-bit precision and backpropagates through it into LoRA adapters3. The authors introduce three pieces: a 4-bit data type called NormalFloat (NF4), double quantisation of the quantisation constants, and paged optimisers to handle memory spikes3. Their headline claim is that a 65B-parameter model can be fine-tuned on a single 48 GB GPU while matching 16-bit fine-tuning performance on the tasks they tested3. Weights are stored in 4 bits but dequantised for computation3.

Diagram: frozen 4-bit base weights hold most of the memory; trainable adapters and optimiser state are small

Diagram: my own.

Compare this with plain quantisation for inference: there the goal is to serve a model cheaply; here it is to train adapters on a model that would otherwise not fit.

What it costs

LoRA is not free quality. Biderman et al. compared LoRA with full fine-tuning on programming and mathematics, and found LoRA substantially underperformed in standard low-rank settings, while preserving the base model’s performance on other tasks better; they also found that full fine-tuning learns weight changes with rank 10 to 100 times higher than typical LoRA settings4. My reading: LoRA is a good fit for style, format and narrow tasks, and a weaker fit for teaching a lot of new skill. Test on your task.

Choosing

  • Single GPU, moderate data, behaviour change: LoRA or QLoRA.
  • Many customers or tasks: separate adapters on one base.
  • Large skill gap, lots of data, strong hardware: consider full tuning, or a higher rank.
  • Merging: merge adapters for the simplest serving, or serve adapters dynamically if your engine supports it (Serving stacks).

Always re-run your eval set on the merged or quantised result, since each step can change behaviour (Building eval sets). Data matters more than the method: see Preparing training data.

Sources

  1. [1]LoRA: Low-Rank Adaptation of Large Language Models — Hu et al. · 2021PARAPHRASED
  2. [2]LoRA: Low-Rank Adaptation of Large Language Models (Microsoft Research page) — Microsoft ResearchSUMMARY OF PARAMETER AND MEMORY REDUCTIONS
  3. [3]QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers, Pagnoni, Holtzman, Zettlemoyer · 2023PARAPHRASED
  4. [4]LoRA Learns Less and Forgets Less — Biderman et al. · 2024PARAPHRASED
  5. [5]LoRA paper breakdown — TheoremPathTYPICAL RANK VALUES, AS SUMMARISED BY A THIRD PARTY

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.