Subscribe
Topic

LoRA

LoRA (low-rank adaptation) makes fine-tuning large models cheap. Posts explain how LoRA and QLoRA work, with the math, cover the DoRA variant, and compare multi-LoRA serving systems such as S-LoRA, Punica and LoRAX.

Articles

1-2
  1. Weight-Decomposed Low-Rank Adaptation (DoRA): How Decoupling Magnitude and Direction Closes the LoRA Gap

    Weight-Decomposed Low-Rank Adaptation (DoRA): How Decoupling Magnitude and Direction Closes the LoRA Gap Parameter-efficient fine-tuning (PEFT) has become the standard operational paradigm for adapting large language models to domain-specific downstream tasks. Among existing PEFT methodologies, Low-Rank Adaptation (LoRA) remains the default implementation across industry and academia due to its minimal parameter footprint and zero inference overhead. However, empirical studies consistently reve

    AI
    8 min read
  2. Low-Rank Adaptation (LoRA): How Matrix Decomposition Made LLM Fine-Tuning Accessible

    Full-parameter fine-tuning of modern foundation models requires substantial compute and memory infrastructure. Adapting an open-weight 70-billion-parameter model using standard 16-bit precision and first-order adaptive optimizers like AdamW demands well over 1 terabyte of GPU memory. Low-Rank Adaptation (LoRA) bypassed this hardware bottleneck by framing task-specific weight updates as low-rank matrix decompositions. By freezing pre-trained weights and training small auxiliary rank decompositio

    AI
    6 min read