Subscribe
Topic

Fine-Tuning

How to fine-tune LLMs, from parameter-efficient methods to the frameworks that run them. Posts compare Unsloth, Axolotl, LLaMA-Factory and Torchtune, and explain LoRA, QLoRA and reference-free preference tuning with SimPO and ORPO.

Articles

1-20
  1. Reference-Free Preference Optimization: Mathematical Foundations of SimPO and ORPO, Length-Normalized Implicit Rewards, Target Margins, and Monolithic Alignment

    Reference-Free Preference Optimization: Mathematical Foundations of SimPO and ORPO, Length-Normalized Implicit Rewards, Target Margins, and Monolithic Alignment Post-training alignment has shifted from complex multi-stage reinforcement learning pipelines toward direct preference optimization paradigms. While Proximal Policy Optimization (PPO) requires maintaining four concurrent models in memory (policy, value, reference, and reward networks), Direct Preference Optimization (DPO) reduced this f

    AI
    13 min read
  2. LLM Fine-Tuning and Post-Training Frameworks in Production: Comparing Unsloth, Axolotl, Torchtune, and LLaMA-Factory

    Post-training has transitioned from a specialized research task into a standard production engineering discipline. As open-weight base models such as Llama 3.1, Qwen 2.5, and DeepSeek-V3 establish competitive baselines, the primary engineering challenge has shifted toward domain adaptation, instruction alignment, and reasoning distillation. However, selecting a post-training framework requires balancing competing architectural trade-offs: low-level kernel fusion, distributed multi-node scaling,

    AI
    6 min read
  3. Low-Rank Adaptation (LoRA) and QLoRA: Mathematical Foundations, Intrinsic Rank Parameterization, NF4 Quantization, and Double Quantization Mechanics

    Low-Rank Adaptation (LoRA) and QLoRA: Mathematical Foundations, Intrinsic Rank Parameterization, NF4 Quantization, and Double Quantization Mechanics Parameter-efficient fine-tuning (PEFT) has become the standard operational methodology for adapting large language models to domain-specific tasks, downstream instruction following, and structured tool use. Full-parameter fine-tuning of frontier architectures requires updating and tracking optimizer states for tens or hundreds of billions of parame

    AI
    11 min read
  4. QLoRA: Mathematical Foundations of 4-Bit NormalFloat Quantization, Double Quantization, and Paged Optimizers

    Fine-tuning large language models in full 16-bit precision requires substantial hardware infrastructure. For a 65-billion parameter model, storing weights in 16-bit brain floating point (BF16) requires 130 GB of VRAM. When optimizing those parameters with 32-bit AdamW, storing gradients (130 GB), first moments (260 GB), second moments (260 GB), and master weights (260 GB) drives the minimum memory footprint beyond 780 GB of VRAM, even before accounting for sequence activations. While Low-Rank A

    AI
    7 min read
  5. Kahneman-Tversky Optimization (KTO): Mathematical Foundations, Prospect Theory, and Binary Signal Alignment in Large Language Models

    Post-training alignment of large language models has long relied on pairwise comparison datasets. Methods such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) assume access to curated pairs $(x, y_w, y_l)$, where a human or automated judge explicitly marks completion $y_w$ as superior to $y_l$ for a given prompt $x$. In production environments, however, collecting paired preferences is logistically complex, expensive, and artificial. Real-world telem

    Explainers
    10 min read
  6. Low-Rank Adaptation (LoRA) and DoRA: Mathematical Foundations, Intrinsic Dimensionality, and Directional Weight Decomposition

    Low-Rank Adaptation (LoRA) and DoRA: Mathematical Foundations, Intrinsic Dimensionality, and Directional Weight Decomposition Full-parameter fine-tuning of large language models presents prohibitive memory requirements during training. Standard full fine-tuning requires updating every weight matrix in a model, demanding storage not only for the model parameters themselves, but also for activation tensors, backward gradients, and first- and second-moment optimizer states. For a 70-billion-parame

    AI
    15 min read
  7. The Self-Taught Reasoner (STaR): How Iterative Rationale Bootstrapping and Rationalization Scale LLM Problem Solving

    Autoregressive language models often fail when forced to predict answers to multi-step reasoning problems in a single forward pass. Intermediate chain-of-thought generation allows models to allocate additional compute to difficult reasoning steps before emitting a final prediction. However, obtaining high-quality intermediate reasoning traces has traditionally presented a difficult trade-off: either rely on small prompt-based few-shot demonstrations that underperform fine-tuned models, or constr

    AI
    4 min read
  8. Identity Preference Optimization (IPO): How Exact Loss Inversion Prevents Overfitting in Direct Alignment

    The post-training alignment of large language models underwent a structural shift with the introduction of Direct Preference Optimization (Rafailov et al., 2023). By reparameterizing the closed-form optimal policy under a Kullback-Leibler (KL) constrained Reinforcement Learning from Human Feedback (RLHF) objective, DPO eliminated the need to fit an explicit reward model or maintain complex actor-critic rollout loops. However, standard DPO introduces a distinct mathematical pathology: under dete

    AI
    7 min read
  9. Embedding Model Fine-Tuning in Production: Hard Negative Mining, Synthetic Data Pipelines, and Contrastive Distillation

    Production retrieval-augmented generation (RAG) and semantic search architectures frequently suffer from domain mismatch when relying on general-purpose embedding models. Off-the-shelf bi-encoders trained on broad web corpora often experience a 15% to 30% degradation in retrieval metrics such as NDCG@10 and MRR@10 when deployed on specialized enterprise corpora, including proprietary codebases, internal API schemas, clinical trials, and technical documentation. While downstream cross-encoder re

    RAG
    8 min read
  10. Retrieval-Augmented Fine-Tuning (RAFT) in Production: Architecture, Synthetic Distractor Pipelines, and Evaluation

    Standard approaches to enterprise domain adaptation typically force a choice between two paradigms: Supervised Fine-Tuning (SFT) or Retrieval-Augmented Generation (RAG). SFT bakes domain knowledge directly into model weights, functioning like a closed-book exam. When facts change or precise source attribution is required, SFT models often hallucinate or fail to incorporate updated context. Conversely, standard RAG operates like an open-book exam without prior preparation. The base model reads re

    AI
    5 min read
  11. Fine-Tuning vs. RAG in Production: Knowledge Injection, Task Adaptation, Latency Economics, and Hybrid Architecture Trade-Offs

    When adapting large language models to domain-specific enterprise workloads, engineering teams face a fundamental architectural choice: modify the model's parametric weights via fine-tuning, or supply dynamic context at inference time via Retrieval-Augmented Generation (RAG). While early discussions often framed this decision as a binary trade-off, empirical evaluations demonstrate that fine-tuning and retrieval serve fundamentally different mathematical and operational functions. Fine-tuning e

    AI
    6 min read
  12. NormalFloat (NF4) and Double Quantization: The Information-Theoretic Foundations of QLoRA

    Fine-tuning large language models under full 16-bit precision is governed by strict memory scaling laws. For a standard 65-billion parameter transformer model, storing weights in 16-bit BrainFloat (BF16) or Float16 (FP16) requires 130 GB of GPU memory. During training with standard first-order adaptive optimizers such as AdamW, each parameter requires an additional 2 bytes for gradients and 8 bytes for FP32 optimizer states (4 bytes for first-moment momentum and 4 bytes for second-moment varianc

    AI
    10 min read
  13. Parameter-Efficient Fine-Tuning Beyond LoRA: How Adapters, Prefix Tuning, Prompt Tuning, and (IA)3 Adapt Frozen LLMs

    Parameter-Efficient Fine-Tuning Beyond LoRA: How Adapters, Prefix Tuning, Prompt Tuning, and (IA)3 Adapt Frozen LLMs While Low-Rank Adaptation (LoRA) and its weight-decomposed variant (DoRA) serve as the prevailing standard for adapting large language models, parameter-efficient fine-tuning (PEFT) encompasses a broader taxonomy of mathematical approaches. The core objective of PEFT is to adapt multi-billion-parameter foundation models to specialized downstream tasks while updating only a fracti

    AI
    8 min read
  14. Weight-Decomposed Low-Rank Adaptation (DoRA): How Decoupling Magnitude and Direction Closes the LoRA Gap

    Weight-Decomposed Low-Rank Adaptation (DoRA): How Decoupling Magnitude and Direction Closes the LoRA Gap Parameter-efficient fine-tuning (PEFT) has become the standard operational paradigm for adapting large language models to domain-specific downstream tasks. Among existing PEFT methodologies, Low-Rank Adaptation (LoRA) remains the default implementation across industry and academia due to its minimal parameter footprint and zero inference overhead. However, empirical studies consistently reve

    AI
    8 min read
  15. Supervised Fine-Tuning in Large Language Models: Loss Masking, Sequence Packing, and Alignment Dynamics

    Pre-training endows a large language model with broad linguistic patterns, world knowledge, and reasoning primitives by predicting the next token across trillions of uncurated web tokens. However, a raw base model remains a document completer rather than an interactive assistant. Given a prompt such as "Explain how a compiler works," a base model is as likely to generate additional exam questions or web navigation headers as it is to answer the query. Supervised Fine-Tuning (SFT), frequently te

    AI
    7 min read
  16. Reference-Free Preference Optimization: How ORPO and SimPO Eliminate Reference Models in LLM Alignment

    Aligning large language models with human preferences has historically relied on two distinct stages after pretraining: Supervised Fine-Tuning (SFT) to establish instruction-following behaviors, followed by reinforcement learning from human feedback (RLHF) or Direct Preference Optimization (DPO) to maximize response quality. While DPO removed the need for explicit reward modeling and complex actor-critic policy loops (such as Proximal Policy Optimization, or PPO), it retained an architectural b

    AI
    7 min read