Low-Rank Adaptation (LoRA) and QLoRA: Parameter-Efficient Fine-Tuning, Matrix Decomposition, and 4-Bit Quantization
Low-Rank Adaptation (LoRA) and QLoRA: Parameter-Efficient Fine-Tuning, Matrix Decomposition, and 4-Bit Quantization Training a large language model from scratch requires massive compute. Adapting a pre-trained model to a downstream task through full fine-tuning requires storing optimizer states, gradients, and activations for every parameter — often multiple terabytes for a 70B model. Low-Rank Adaptation (LoRA) and its quantized successor QLoRA changed that calculus: they make task-specific ada
1 min