llms.blog
LoRA6 articles

LoRA

Articles

  • Low-Rank Adaptation (LoRA) and QLoRA: Parameter-Efficient Fine-Tuning, Matrix Decomposition, and 4-Bit Quantization

    Low-Rank Adaptation (LoRA) and QLoRA: Parameter-Efficient Fine-Tuning, Matrix Decomposition, and 4-Bit Quantization Training a large language model from scratch requires massive compute. Adapting a pre-trained model to a downstream task through full fine-tuning requires storing optimizer states, gradients, and activations for every parameter — often multiple terabytes for a 70B model. Low-Rank Adaptation (LoRA) and its quantized successor QLoRA changed that calculus: they make task-specific ada

    1 min

  • Low-Rank Adaptation (LoRA) and QLoRA: Mathematical Foundations, Intrinsic Rank Dynamics, 4-Bit NormalFloat Quantization, and Memory-Efficient Fine-Tuning

    Fine-tuning large language models on custom datasets presents a significant hardware challenge. While running inference on a 70-billion parameter model requires only the model weights in memory, full-parameter fine-tuning (FPFT) demands an order of magnitude more resources. During standard 16-bit mixed-precision training with optimizers such as AdamW, each parameter requires 2 bytes for the static weight, 2 bytes for the gradient, 4 bytes for the 32-bit master weight copy, and 8 bytes for the fi

    1 min

  • Low-Rank Adaptation (LoRA) and QLoRA: Mathematical Foundations, Intrinsic Rank Dynamics, NF4 Quantization, and Parameter-Efficient Fine-Tuning

    Full fine-tuning of large language models requires updating every parameter matrix across all transformer blocks. In production architectures spanning tens to hundreds of billions of parameters, the computational and memory footprint of updating billions of weights with first-order and second-order optimizer states becomes prohibitive. Low-Rank Adaptation (LoRA) and its quantized counterpart QLoRA provide mathematically grounded parameter-efficient fine-tuning (PEFT) frameworks. By decomposing

    1 min

  • Multi-LoRA Serving in Production: Comparing S-LoRA, Punica, LoRAX, and vLLM Multi-LoRA Architecture, Batched SGMV Kernels, Paged Adapter Memory, and Co-Location Economics

    Fine-tuning large language models on domain-specific corpora, proprietary workflows, and per-tenant datasets has become a standard enterprise practice. Deploying hundreds or thousands of distinct task-specific models as full weight replicas creates unsustainable infrastructure costs. A 70-billion-parameter base model in 16-bit precision requires approximately 140 GB of high-bandwidth memory (HBM) across two to four high-end GPUs. Serving 500 specialized models as isolated instances would require

    1 min

  • Weight-Decomposed Low-Rank Adaptation (DoRA): How Decoupling Magnitude and Direction Closes the LoRA Gap

    Weight-Decomposed Low-Rank Adaptation (DoRA): How Decoupling Magnitude and Direction Closes the LoRA Gap Parameter-efficient fine-tuning (PEFT) has become the standard operational paradigm for adapting large language models to domain-specific downstream tasks. Among existing PEFT methodologies, Low-Rank Adaptation (LoRA) remains the default implementation across industry and academia due to its minimal parameter footprint and zero inference overhead. However, empirical studies consistently reve

    1 min

  • Low-Rank Adaptation (LoRA): How Matrix Decomposition Made LLM Fine-Tuning Accessible

    Full-parameter fine-tuning of modern foundation models requires substantial compute and memory infrastructure. Adapting an open-weight 70-billion-parameter model using standard 16-bit precision and first-order adaptive optimizers like AdamW demands well over 1 terabyte of GPU memory. Low-Rank Adaptation (LoRA) bypassed this hardware bottleneck by framing task-specific weight updates as low-rank matrix decompositions. By freezing pre-trained weights and training small auxiliary rank decompositio

    1 min