Alignment22 articles

Alignment

Articles

Anthropic Demonstrates Automated Alignment Researchers That Outperform Human Safety Teams
Anthropic has published research demonstrating that autonomous AI agents can systematically discover, implement, and validate post-training methods to mitigate safety and alignment failures in language models. The report, authored by Anthropic Fellow Chen Yueh-Han and colleagues, evaluates an automated research loop that closed between 26% and 96% of the safety gap across ten distinct alignment failure categories without degrading baseline model capabilities. The findings provide empirical evid
August 28, 20261 min
Direct Preference Optimization (DPO): Mathematical Foundations, Implicit Reward Reparameterization, Bradley-Terry Policy Derivation, and Closed-Form Alignment Dynamics
Direct Preference Optimization (DPO): Mathematical Foundations, Implicit Reward Reparameterization, Bradley-Terry Policy Derivation, and Closed-Form Alignment Dynamics Post-training alignment is the critical bridge that transforms raw auto-regressive language models into coherent, steerable, and safe assistants. For several years, the standard approach to preference alignment relied on Reinforcement Learning from Human Feedback (RLHF) executed via Proximal Policy Optimization (PPO). While conce
August 28, 20261 min
Kahneman-Tversky Optimization (KTO): Mathematical Foundations, Prospect Theory Utility, Binary Feedback Alignment, and Loss Aversion Dynamics
Kahneman-Tversky Optimization (KTO): Mathematical Foundations, Prospect Theory Utility, Binary Feedback Alignment, and Loss Aversion Dynamics Aligning large language models with human expectations has historically relied on pairwise preference data. Under traditional Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), models learn from paired tuples where one completion is preferred over another for a given prompt. While mathematically grounded in the Bra
August 28, 20261 min
Simple Preference Optimization (SimPO): Mathematical Foundations, Reference-Free Implicit Reward, Length Normalization, and Target Margin Dynamics
Simple Preference Optimization (SimPO): Mathematical Foundations, Reference-Free Implicit Reward, Length Normalization, and Target Margin Dynamics Post-training preference alignment has become the definitive step in transforming raw pretrained large language models into instruction-following assistants. While Reinforcement Learning from Human Feedback (RLHF) via Proximal Policy Optimization (PPO) established the initial standard, its requirement to maintain actor, critic, reference, and reward
August 28, 20261 min
Contrastive Preference Optimization (CPO): Mathematical Foundations, Direct Likelihood Regularization, Out-of-Distribution Error Bounds, and Non-Degenerate Policy Alignment
Supervised fine-tuning (SFT) and preference alignment represent the standard two-stage training paradigm for post-training large language models. While SFT establishes foundational task adherence by minimizing cross-entropy loss against target references, it suffers from two structural limitations: it cannot surpass the quality ceiling of its training demonstrations, and it lacks an explicit mechanism to penalize near-correct outputs that contain subtle hallucinations or omissions. Direct Prefe
August 28, 20261 min
Reference-Free Preference Optimization: Mathematical Foundations of SimPO and ORPO, Length-Normalized Implicit Rewards, Target Margins, and Monolithic Alignment
Reference-Free Preference Optimization: Mathematical Foundations of SimPO and ORPO, Length-Normalized Implicit Rewards, Target Margins, and Monolithic Alignment Post-training alignment has shifted from complex multi-stage reinforcement learning pipelines toward direct preference optimization paradigms. While Proximal Policy Optimization (PPO) requires maintaining four concurrent models in memory (policy, value, reference, and reward networks), Direct Preference Optimization (DPO) reduced this f
August 28, 20261 min
Direct Preference Optimization (DPO): Mathematical Foundations, Implicit Reward Derivation, Closed-Form Bradley-Terry Equivalence, and Reference Policy Regularization
Direct Preference Optimization (DPO): Mathematical Foundations, Implicit Reward Derivation, Closed-Form Bradley-Terry Equivalence, and Reference Policy Regularization Post-training preference alignment is the central mechanism for transforming pretrained base large language models into instruction-following, steerable, and safe conversational agents. Historically, the dominant paradigm for preference alignment has been Reinforcement Learning from Human Feedback (Christiano et al., 2017; Ouyang
August 26, 20261 min
Odds Ratio Preference Optimization (ORPO): Mathematical Foundations, Monolithic SFT-Alignment, and Reference-Free Odds Penalties
Post-training large language models typically relies on a decoupled, two-stage pipeline: Supervised Fine-Tuning (SFT) on curated instruction-response pairs, followed by Preference Alignment using algorithms such as Reinforcement Learning from Human Feedback (PPO) or Direct Preference Optimization (DPO). While effective, this multi-stage paradigm introduces substantial operational and computational friction. First, DPO and PPO require maintaining auxiliary reference models in GPU memory to preve
August 26, 20261 min
Simple Preference Optimization (SimPO): Mathematical Foundations, Length-Normalized Implicit Rewards, and Reference-Free Margin Alignment
Aligning large language models with human preferences has evolved rapidly from complex multi-stage reinforcement learning pipelines toward direct, offline optimization objectives. While Direct Preference Optimization (DPO) eliminated the requirement for training a separate reward model and executing online policy rollouts, it introduced architectural and operational challenges of its own: the necessity of maintaining an active reference policy in GPU memory, an unnormalized log-probability formu
August 25, 20261 min
Direct Preference Optimization: Mathematical Derivation, Implicit Reward Formulation, and the Mechanics of RL-Free Alignment
Direct Preference Optimization: Mathematical Derivation, Implicit Reward Formulation, and the Mechanics of RL-Free Alignment Aligning autoregressive large language models with human preferences has traditionally relied on Reinforcement Learning from Human Feedback (RLHF). In the standard formulation popularized by InstructGPT and related post-training regimes, alignment requires a multi-stage pipeline: supervised fine-tuning (SFT), training a separate reward model on pairwise comparison data, a
August 25, 20261 min
Self-Rewarding Language Models: How Iterative DPO and LLM-as-a-Judge Form Autonomous Self-Alignment Loops
Standard post-training alignment pipelines rely on frozen reward models trained on static human feedback datasets. While Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) effectively steer model outputs toward human preferences, they face a fundamental scalability bottleneck: human annotators cannot evaluate superhuman reasoning or generate labels at the scale required for continuous self-improvement. Self-Rewarding Language Models, introduced by Meta AI
August 24, 20261 min
Rejection Sampling Fine-Tuning in Large Language Models: How Best-of-N Filtering, Reward Oracles, and Distillation Align Neural Policies
Rejection Sampling Fine-Tuning: How Filtering Model Outputs by Reward Optimizes Alignment Without Policy Gradient Instability Post-training alignment has become a defining phase in modern large language model development. While supervised fine-tuning (SFT) teaches a model to follow instructions and adopt structured formats, aligning model behavior with human preferences, safety criteria, and domain accuracy requires optimizing against reward signals. Historically, this optimization has been ap
August 23, 20261 min
Identity Preference Optimization (IPO): How Exact Loss Inversion Prevents Overfitting in Direct Alignment
The post-training alignment of large language models underwent a structural shift with the introduction of Direct Preference Optimization (Rafailov et al., 2023). By reparameterizing the closed-form optimal policy under a Kullback-Leibler (KL) constrained Reinforcement Learning from Human Feedback (RLHF) objective, DPO eliminated the need to fit an explicit reward model or maintain complex actor-critic rollout loops. However, standard DPO introduces a distinct mathematical pathology: under dete
August 23, 20261 min
Reward Model Overoptimization in Large Language Models: How Goodhart's Law, Proxy Exploitation, and KL Drift Degrade Alignment
Post-training alignment of large language models relies on optimizing a policy toward objectives defined by human intent and preferences. Because querying human evaluators during every step of continuous reinforcement learning or high-throughput rejection sampling is computationally and logistically infeasible, alignment workflows construct a parameterised proxy reward model. Trained on pairwise preference datasets through formulations such as the Bradley-Terry model, this proxy acts as a surrog
August 22, 20261 min
Self-Play Fine-Tuning (SPIN): How Iterative Games Against Historical Checkpoints Eliminate Preference Labels
Self-Play Fine-Tuning (SPIN): How Iterative Games Against Historical Checkpoints Eliminate Preference Labels Supervised fine-tuning (SFT) remains the standard entry point for adapting foundation language models to downstream instructions. By training on curated prompt-response pairs via token-level cross-entropy loss, SFT forces a model to mirror reference demonstrations. However, standard maximum likelihood estimation (MLE) suffers from exposure bias and covariate shift. During inference, the
August 20, 20261 min
Rejection Sampling Fine-Tuning: How Filtering Model Outputs by Reward Replaced RLHF Complexity
Rejection Sampling Fine-Tuning: How Filtering Model Outputs by Reward Replaced RLHF Complexity Rejection sampling fine-tuning (RAFT) has emerged as the practical workhorse of LLM alignment. While PPO-based RLHF dominated early literature, production systems from Llama 2 to DeepSeek-R1 rely on a simpler loop: generate multiple completions per prompt, score them with a reward model, keep the best, and fine-tune on the filtered data. The technique converts the reinforcement learning problem into s
August 20, 20261 min
Kahneman-Tversky Optimization: Aligning LLMs with Prospect Theory and Binary Feedback
Alignment of large language models has traditionally centered on preference learning. Methods such as Reinforcement Learning from Human Feedback (Christiano et al., 2017), Direct Preference Optimization (Rafailov et al., 2023), and Identity Preference Optimization (Azar et al., 2023) require training data formatted as pairs of candidate responses $(x, y_w, y_l)$ generated for the exact same prompt $x$, where $y_w$ is preferred over $y_l$. In real-world production environments, paired preference
August 19, 20261 min
Reference-Free Preference Optimization: How ORPO and SimPO Eliminate Reference Models in LLM Alignment
Aligning large language models with human preferences has historically relied on two distinct stages after pretraining: Supervised Fine-Tuning (SFT) to establish instruction-following behaviors, followed by reinforcement learning from human feedback (RLHF) or Direct Preference Optimization (DPO) to maximize response quality. While DPO removed the need for explicit reward modeling and complex actor-critic policy loops (such as Proximal Policy Optimization, or PPO), it retained an architectural b
August 19, 20261 min
Constitutional AI and RLAIF: How Natural Language Principles and Automated Critiques Scale LLM Alignment
Early alignment frameworks for large language models relied almost entirely on Reinforcement Learning from Human Feedback (RLHF). While RLHF transformed raw base models into usable assistants, the approach faces structural scaling bottlenecks. Collecting tens of thousands of high-quality human preference annotations is slow, expensive, and logistically complex. Furthermore, human annotators frequently disagree on nuanced safety boundaries, suffer psychological fatigue when reviewing harmful outp
August 18, 20261 min
Reinforcement Learning from Human Feedback (RLHF): How Reward Models, PPO, and KL Penalties Align LLMs
Pre-training large language models on internet-scale text corpora equips them with general linguistic patterns, world knowledge, and broad reasoning heuristics. However, pre-training optimizes next-token prediction: $\mathbb{E}_{x \sim \mathcal{D}} [\log P_\theta(x_t \mid x_{<t})]$. A base model trained purely on next-token prediction reflects the entirety of its web corpus, reproducing hallucinations, toxic phrasing, incorrect code, and unhelpful conversational patterns. To transform an uncons
August 18, 20261 min