Low-Rank Adaptation (LoRA) and QLoRA: Mathematical Foundations, Intrinsic Rank Dynamics, 4-Bit NormalFloat Quantization, and Memory-Efficient Fine-Tuning
Fine-tuning large language models on custom datasets presents a significant hardware challenge. While running inference on a 70-billion parameter model requires only the model weights in memory, full-parameter fine-tuning (FPFT) demands an order of magnitude more resources. During standard 16-bit mixed-precision training with optimizers such as AdamW, each parameter requires 2 bytes for the static weight, 2 bytes for the gradient, 4 bytes for the 32-bit master weight copy, and 8 bytes for the fi
1 min