Low-Rank Adaptation (LoRA): How Matrix Decomposition Made LLM Fine-Tuning Accessible
Full-parameter fine-tuning of modern foundation models requires substantial compute and memory infrastructure. Adapting an open-weight 70-billion-parameter model using standard 16-bit precision and first-order adaptive optimizers like AdamW demands well over 1 terabyte of GPU memory. Low-Rank Adaptation (LoRA) bypassed this hardware bottleneck by framing task-specific weight updates as low-rank matrix decompositions. By freezing pre-trained weights and training small auxiliary rank decompositio
1 min