post-training12 articles

post-training

Articles

Direct Preference Optimization (DPO): Mathematical Foundations, Implicit Reward Reparameterization, Bradley-Terry Policy Derivation, and Closed-Form Alignment Dynamics
Direct Preference Optimization (DPO): Mathematical Foundations, Implicit Reward Reparameterization, Bradley-Terry Policy Derivation, and Closed-Form Alignment Dynamics Post-training alignment is the critical bridge that transforms raw auto-regressive language models into coherent, steerable, and safe assistants. For several years, the standard approach to preference alignment relied on Reinforcement Learning from Human Feedback (RLHF) executed via Proximal Policy Optimization (PPO). While conce
August 28, 20261 min
Contrastive Preference Optimization (CPO): Mathematical Foundations, Direct Likelihood Regularization, Out-of-Distribution Error Bounds, and Non-Degenerate Policy Alignment
Supervised fine-tuning (SFT) and preference alignment represent the standard two-stage training paradigm for post-training large language models. While SFT establishes foundational task adherence by minimizing cross-entropy loss against target references, it suffers from two structural limitations: it cannot surpass the quality ceiling of its training demonstrations, and it lacks an explicit mechanism to penalize near-correct outputs that contain subtle hallucinations or omissions. Direct Prefe
August 28, 20261 min
LLM Post-Training Reinforcement Learning Frameworks in Production: Comparing verl, OpenRLHF, TRL, and Ray PPO Architecture, Rollout Scheduling, 3D Parallelism, and Serving Economics
Large language model post-training has undergone a fundamental shift. While supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) operate on static offline datasets with predictable forward-backward compute passes, reasoning models and reinforcement learning with verifiable rewards (RLVR) depend on high-throughput online rollouts. In reasoning-focused RL pipelines, generating candidate completions accounts for 70% to 85% of total iteration time. This creates an architectural dil
August 28, 20261 min
Reference-Free Preference Optimization: Mathematical Foundations of SimPO and ORPO, Length-Normalized Implicit Rewards, Target Margins, and Monolithic Alignment
Reference-Free Preference Optimization: Mathematical Foundations of SimPO and ORPO, Length-Normalized Implicit Rewards, Target Margins, and Monolithic Alignment Post-training alignment has shifted from complex multi-stage reinforcement learning pipelines toward direct preference optimization paradigms. While Proximal Policy Optimization (PPO) requires maintaining four concurrent models in memory (policy, value, reference, and reward networks), Direct Preference Optimization (DPO) reduced this f
August 28, 20261 min
Proximal Policy Optimization (PPO) in LLM Post-Training: Mathematical Foundations of Clipped Surrogate Objectives, Generalized Advantage Estimation, Value Network Architectures, and Adaptive KL Penalties
Proximal Policy Optimization (PPO) in LLM Post-Training: Mathematical Foundations of Clipped Surrogate Objectives, Generalized Advantage Estimation, Value Network Architectures, and Adaptive KL Penalties Reinforcement Learning from Human Feedback (RLHF) established modern alignment for instruction-following foundation models. While pre-training optimizes autoregressive language models to minimize next-token prediction loss over massive text corpora, standard cross-entropy loss cannot directly o
August 27, 20261 min
Deep Cogito Raises 3M Series A to Scale Post-Training and Iterated Distillation
San Francisco AI research startup Deep Cogito has raised a $43 million Series A round to expand its post-training systems and reinforcement learning infrastructure for open-weight foundation models. The round was led by TQ Ventures, with participation from Benchmark, Nexus Venture Partners, Atreides Management, South Park Commons, and enterprise cloud security provider Zscaler, which acts as both a commercial customer and strategic investor. The financing brings total capital raised by Deep Cog
August 26, 20261 min
Direct Preference Optimization (DPO): Mathematical Foundations, Implicit Reward Derivation, Closed-Form Bradley-Terry Equivalence, and Reference Policy Regularization
Direct Preference Optimization (DPO): Mathematical Foundations, Implicit Reward Derivation, Closed-Form Bradley-Terry Equivalence, and Reference Policy Regularization Post-training preference alignment is the central mechanism for transforming pretrained base large language models into instruction-following, steerable, and safe conversational agents. Historically, the dominant paradigm for preference alignment has been Reinforcement Learning from Human Feedback (Christiano et al., 2017; Ouyang
August 26, 20261 min
Odds Ratio Preference Optimization (ORPO): Mathematical Foundations, Monolithic SFT-Alignment, and Reference-Free Odds Penalties
Post-training large language models typically relies on a decoupled, two-stage pipeline: Supervised Fine-Tuning (SFT) on curated instruction-response pairs, followed by Preference Alignment using algorithms such as Reinforcement Learning from Human Feedback (PPO) or Direct Preference Optimization (DPO). While effective, this multi-stage paradigm introduces substantial operational and computational friction. First, DPO and PPO require maintaining auxiliary reference models in GPU memory to preve
August 26, 20261 min
Synthetic Data Pipelines in Production LLM Post-Training: Architecture, Prompt Evolution, Quality Filtering, and Contamination Control
Synthetic Data Pipelines in Production LLM Post-Training: Architecture, Prompt Evolution, Quality Filtering, and Contamination Control Scaling supervised fine-tuning (SFT) and preference alignment (DPO, PPO, GRPO) through human annotation faces severe economic and operational constraints. Human annotation costs between $5.00 and $50.00 per complex instruction-response trajectory, exhibits significant variance across labeler cohorts, and scales linearly with dataset volume. The LIMA study by Zho
August 25, 20261 min
Self-Rewarding Language Models: How Iterative DPO and LLM-as-a-Judge Form Autonomous Self-Alignment Loops
Standard post-training alignment pipelines rely on frozen reward models trained on static human feedback datasets. While Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) effectively steer model outputs toward human preferences, they face a fundamental scalability bottleneck: human annotators cannot evaluate superhuman reasoning or generate labels at the scale required for continuous self-improvement. Self-Rewarding Language Models, introduced by Meta AI
August 24, 20261 min
Distributed RLHF Frameworks in Production: Comparing OpenRLHF, verl, and TRL Architecture, Ray Resource Scheduling, and Rollout-Training Co-Location
Post-training alignment has shifted from offline preference tuning to large-scale, online reinforcement learning. Modern post-training loops for reasoning models, agentic workflows, and conversational alignment require coordinating multiple distinct neural network roles simultaneously. Under standard Proximal Policy Optimization (PPO), an RL infrastructure pipeline must manage up to four distinct model instances: the Actor (the active policy undergoing gradient updates), the Critic (the value mo
August 23, 20261 min
Rejection Sampling Fine-Tuning: How Filtering Model Outputs by Reward Replaced RLHF Complexity
Rejection Sampling Fine-Tuning: How Filtering Model Outputs by Reward Replaced RLHF Complexity Rejection sampling fine-tuning (RAFT) has emerged as the practical workhorse of LLM alignment. While PPO-based RLHF dominated early literature, production systems from Llama 2 to DeepSeek-R1 rely on a simpler loop: generate multiple completions per prompt, score them with a reward model, keep the best, and fine-tune on the filtered data. The technique converts the reinforcement learning problem into s
August 20, 20261 min