Contrastive Preference Optimization (CPO): Mathematical Foundations, Direct Likelihood Regularization, Out-of-Distribution Error Bounds, and Non-Degenerate Policy Alignment
Supervised fine-tuning (SFT) and preference alignment represent the standard two-stage training paradigm for post-training large language models. While SFT establishes foundational task adherence by minimizing cross-entropy loss against target references, it suffers from two structural limitations: it cannot surpass the quality ceiling of its training demonstrations, and it lacks an explicit mechanism to penalize near-correct outputs that contain subtle hallucinations or omissions. Direct Prefe







