If you have downloaded a custom AI image style or a small add-on file that turns a general model into a specialist, it may have been a LoRA. The name stands for Low-Rank Adaptation. It comes from a 2021 Microsoft research paper, and Hugging Face's PEFT library supports it as a way to customize a large AI model without retraining the whole thing.
This explainer covers the problem LoRA solves, how it works in plain terms, what the original paper measured, how the 2023 QLoRA paper pushed the idea onto a single GPU, and where the method has limits. Every number below comes from the papers and documentation linked in the text.
The problem: fine-tuning a giant model is expensive
A large language model is trained once, at great cost, on a large body of general text. To make it good at one job, such as writing SQL queries or summarizing chats, teams "fine-tune" it: they keep training it on examples of that job.
Classic fine-tuning updates every number (every parameter) in the model. The LoRA paper by Edward Hu and colleagues at Microsoft explains why that becomes a problem at scale. Using GPT-3 175B as the example, the authors note that each fine-tuned copy has as many parameters as the original model, about 175 billion. Storing and serving a separate full copy for every task is, in their words, "prohibitively expensive."
People had tried cheaper options before. The paper groups them into two families and names a weakness in each:
- Adapter layers add small extra layers inside the model. The extra layers have to run one after another at answer time. The paper measured that this can slow answers from GPT-2 medium by more than 30 percent for short, single-request inputs.
- Prefix and prompt tuning learn special input tokens instead of changing the model. The paper reports that prefix tuning is difficult to optimize, and these methods use up part of the model's usable input length.
LoRA was designed to avoid both costs.
The core idea in one sentence
LoRA freezes the original model and trains a small "patch" next to some of its weight matrices, then adds the patch's output to the original output.
Inside a transformer, much of the work happens in big grids of numbers called weight matrices. Fine-tuning changes those grids. The LoRA authors bet that the change needed for a new task is simple, even when the matrix itself is large. In math terms, they hypothesized that the update has a low "intrinsic rank," building on earlier research showing that large models can learn new tasks within a much smaller space than their full size suggests.
So instead of learning a full-size change, LoRA learns the change as the product of two thin matrices, called A and B in the paper. If the original matrix is, say, thousands of rows by thousands of columns, A and B might each have only a handful of rows or columns. That handful is the "rank," written r. The paper found that for GPT-3 175B, a rank as low as one or two could be enough, even though the full matrix dimension is 12,288.
How it works, step by step
- Freeze the base model. None of the original weights change during training.
- Attach two small matrices to selected layers. In the original paper, these were mostly the query and value projections in the attention blocks.
- Start the patch at zero. The paper initializes A with random values and B with zeros, so the product starts at zero and the model behaves exactly like the base model at step one.
- Train only A and B. Gradients flow through the frozen model, but only the small matrices are updated.
- Scale the patch. The output of the patch is multiplied by a constant (alpha divided by r) so that changing the rank does not force you to re-tune everything else.
- Optionally merge for deployment. Because the patch is just a matrix product, you can add it straight into the original weights once training is done. The merged model runs at the same speed as the original.
That last step is a big part of why LoRA spread. The paper states that merging means LoRA adds "no additional inference latency" compared with a fully fine-tuned model. Hugging Face's PEFT documentation describes the same thing in practice: its merge function turns base model plus adapter into a single standalone model.
What the original paper measured
The LoRA paper tested the method on RoBERTa, DeBERTa, GPT-2 and GPT-3 175B, across language understanding benchmarks such as GLUE and generation tasks including WikiSQL (turning questions into SQL) and SAMSum (summarizing conversations). The headline results, as reported by the authors:
- Far fewer trainable parameters. Compared with GPT-3 175B fine-tuned with the Adam optimizer, LoRA cut the number of trainable parameters by 10,000 times. Trainable parameters could be as small as 0.01 percent of the full model.
- Less GPU memory. Training memory on GPT-3 175B dropped from 1.2 TB to 350 GB, roughly a 3x reduction, because the optimizer no longer needs to track state for the frozen weights.
- Tiny checkpoints. With rank 4 and only the query and value matrices adapted, the saved task-specific file shrank from 350 GB to about 35 MB.
- Faster training. The authors observed a 25 percent training speedup on GPT-3 175B versus full fine-tuning.
- Comparable quality. LoRA performed "on-par or better than fine-tuning" in model quality on the four model families tested.
The checkpoint result has a practical twist worth spelling out. You still need the full 350 GB base model to run anything. But the paper points out that storing 100 adapted versions takes about 354 GB (one base model plus 100 small patches) instead of about 35 TB for 100 full copies.
Why "low rank" works at all
It is fair to ask how a patch that small can teach a model anything. The paper's analysis of the learned updates gives a partial answer. The authors found that the LoRA update correlates with the original weights more than a random matrix does, so it "amplifies some features that are already in" the model. But it does not repeat the model's strongest directions. Instead it boosts directions that are "important but not emphasized" in the original weights, and by a large factor: about 21.5 times at rank 4. The authors suggest these are task-specific features the model learned in pretraining but did not stress.
A plain way to put it: the base model already holds much of what a task needs. A LoRA patch turns up the volume on the parts that matter for this job.
The paper also notes that LoRA roughly generalizes full fine-tuning. If you apply it to every weight matrix and raise the rank to match the full matrix, you are back to something close to training the whole model.
QLoRA: fine-tuning a 65B model on one GPU
LoRA cut the cost of the trainable part. But the frozen base model still had to sit in GPU memory at full precision. The QLoRA paper from Tim Dettmers and colleagues at the University of Washington, published in 2023, attacked that remaining cost.
QLoRA stores the frozen base model in 4-bit form (a compressed number format) and trains ordinary LoRA adapters on top. The paper's numbers:
- Regular 16-bit fine-tuning of a 65B-parameter LLaMA model needs more than 780 GB of GPU memory. QLoRA brings that under 48 GB, enough for a single professional GPU.
- The authors report this works "without degrading the runtime or predictive performance" compared with a 16-bit fully fine-tuned baseline on the benchmarks they tested.
- Their best model family, Guanaco, reached 99.3 percent of ChatGPT's score on the Vicuna benchmark after 24 hours of fine-tuning on a single GPU, as judged by GPT-4.
QLoRA added three techniques to make this work:
- 4-bit NormalFloat (NF4), a number format designed for the bell-curve shape that trained model weights usually have.
- Double quantization, which also compresses the small scaling constants used by the compression step, saving about 0.37 bits per parameter, or roughly 3 GB on a 65B model.
- Paged optimizers, which use NVIDIA unified memory to spill optimizer data to CPU RAM when the GPU briefly runs out.
QLoRA also corrected a common habit. The authors found that applying LoRA only to the query and value matrices, the original paper's default, did not match full fine-tuning on large models. Putting adapters on all linear layers of the transformer block was needed. They also found that, once adapters covered all layers, the rank r did not noticeably affect results in their tests.
Two other QLoRA findings are useful for anyone planning a fine-tune. Data quality mattered far more than size: a 9,000-example dataset (OASST1) beat a 450,000-example subset of FLAN v2 on chatbot performance. And strong scores on the MMLU knowledge benchmark did not imply strong chatbot scores, or the other way around.
Where you see LoRA today
- Custom language models. Hugging Face's PEFT library implements LoRA as a standard option: you load a base model, set a rank and target layers in a config, wrap the model, and train as usual, per its LoRA guide. The QLoRA authors also integrated their methods into the Hugging Face transformers stack. For tools that wrap these steps, see our comparison of fine-tuning frameworks.
- Image models. The PEFT documentation notes that, while the original paper focused on language models, the technique applies to any dense layer, including in diffusion image models, and it points to a DreamBooth-with-LoRA guide. It also mentions Stable Diffusion WebUI users switching LoRAs on and off over a base model.
- Many tasks on one model. Because adapters are small and swappable, a single base model can serve many customized versions. The LoRA paper describes swapping adapters "on the fly" on machines that keep the base weights in GPU memory.
Limits and trade-offs
LoRA is not free. The papers themselves flag several limits:
- Batching mixed tasks is awkward. The LoRA paper notes that if you merge an adapter into the weights to avoid latency, you cannot easily batch requests for different tasks in one pass. Keeping adapters unmerged allows mixing, at some cost in speed.
- Defaults can underperform. QLoRA showed that the original attention-only setup fell short of full fine-tuning on large models until adapters covered all layers.
- You still need the base model. The adapter is small, but running it requires the full base model in memory, compressed or not.
- Scope of evidence. The QLoRA authors state they did not establish that QLoRA matches full 16-bit fine-tuning at the 33B and 65B scales, because full fine-tuning at that size was too expensive to run as a baseline. They also caution that chatbot benchmarks and GPT-4-as-judge scores have known biases.
Why it matters
Full fine-tuning of a model the size of GPT-3 175B means training every parameter and storing a full copy per task, which the LoRA paper calls "prohibitively expensive." LoRA showed that a small, mergeable patch could match or beat full fine-tuning on the tasks tested, and QLoRA showed that a 65B model could be fine-tuned on a single 48 GB GPU. The QLoRA authors call that "a significant shift in accessibility of LLM finetuning." For the math behind the method, including rank, scaling and weight merging, see our technical guide to how LoRA works.
