PEFT6 articles

PEFT

Articles

Low-Rank Adaptation (LoRA) and QLoRA: Mathematical Foundations, Intrinsic Rank Parameterization, NF4 Quantization, and Double Quantization Mechanics
Low-Rank Adaptation (LoRA) and QLoRA: Mathematical Foundations, Intrinsic Rank Parameterization, NF4 Quantization, and Double Quantization Mechanics Parameter-efficient fine-tuning (PEFT) has become the standard operational methodology for adapting large language models to domain-specific tasks, downstream instruction following, and structured tool use. Full-parameter fine-tuning of frontier architectures requires updating and tracking optimizer states for tens or hundreds of billions of parame
August 27, 20261 min
Low-Rank Adaptation (LoRA) and QLoRA: Mathematical Foundations, Intrinsic Rank Dynamics, 4-Bit NormalFloat Quantization, and Memory-Efficient Fine-Tuning
Fine-tuning large language models on custom datasets presents a significant hardware challenge. While running inference on a 70-billion parameter model requires only the model weights in memory, full-parameter fine-tuning (FPFT) demands an order of magnitude more resources. During standard 16-bit mixed-precision training with optimizers such as AdamW, each parameter requires 2 bytes for the static weight, 2 bytes for the gradient, 4 bytes for the 32-bit master weight copy, and 8 bytes for the fi
August 27, 20261 min
Low-Rank Adaptation (LoRA) and DoRA: Mathematical Foundations, Intrinsic Dimensionality, and Directional Weight Decomposition
Low-Rank Adaptation (LoRA) and DoRA: Mathematical Foundations, Intrinsic Dimensionality, and Directional Weight Decomposition Full-parameter fine-tuning of large language models presents prohibitive memory requirements during training. Standard full fine-tuning requires updating every weight matrix in a model, demanding storage not only for the model parameters themselves, but also for activation tensors, backward gradients, and first- and second-moment optimizer states. For a 70-billion-parame
August 25, 20261 min
Parameter-Efficient Fine-Tuning Beyond LoRA: How Adapters, Prefix Tuning, Prompt Tuning, and (IA)3 Adapt Frozen LLMs
Parameter-Efficient Fine-Tuning Beyond LoRA: How Adapters, Prefix Tuning, Prompt Tuning, and (IA)3 Adapt Frozen LLMs While Low-Rank Adaptation (LoRA) and its weight-decomposed variant (DoRA) serve as the prevailing standard for adapting large language models, parameter-efficient fine-tuning (PEFT) encompasses a broader taxonomy of mathematical approaches. The core objective of PEFT is to adapt multi-billion-parameter foundation models to specialized downstream tasks while updating only a fracti
August 20, 20261 min
Weight-Decomposed Low-Rank Adaptation (DoRA): How Decoupling Magnitude and Direction Closes the LoRA Gap
Weight-Decomposed Low-Rank Adaptation (DoRA): How Decoupling Magnitude and Direction Closes the LoRA Gap Parameter-efficient fine-tuning (PEFT) has become the standard operational paradigm for adapting large language models to domain-specific downstream tasks. Among existing PEFT methodologies, Low-Rank Adaptation (LoRA) remains the default implementation across industry and academia due to its minimal parameter footprint and zero inference overhead. However, empirical studies consistently reve
August 20, 20261 min
Low-Rank Adaptation (LoRA): How Matrix Decomposition Made LLM Fine-Tuning Accessible
Full-parameter fine-tuning of modern foundation models requires substantial compute and memory infrastructure. Adapting an open-weight 70-billion-parameter model using standard 16-bit precision and first-order adaptive optimizers like AdamW demands well over 1 terabyte of GPU memory. Low-Rank Adaptation (LoRA) bypassed this hardware bottleneck by framing task-specific weight updates as low-rank matrix decompositions. By freezing pre-trained weights and training small auxiliary rank decompositio
August 18, 20261 min

PEFT

Articles

Low-Rank Adaptation (LoRA) and QLoRA: Mathematical Foundations, Intrinsic Rank Parameterization, NF4 Quantization, and Double Quantization Mechanics

Low-Rank Adaptation (LoRA) and QLoRA: Mathematical Foundations, Intrinsic Rank Dynamics, 4-Bit NormalFloat Quantization, and Memory-Efficient Fine-Tuning

Low-Rank Adaptation (LoRA) and DoRA: Mathematical Foundations, Intrinsic Dimensionality, and Directional Weight Decomposition

Parameter-Efficient Fine-Tuning Beyond LoRA: How Adapters, Prefix Tuning, Prompt Tuning, and (IA)3 Adapt Frozen LLMs

Weight-Decomposed Low-Rank Adaptation (DoRA): How Decoupling Magnitude and Direction Closes the LoRA Gap

Low-Rank Adaptation (LoRA): How Matrix Decomposition Made LLM Fine-Tuning Accessible