Chatbots do not become helpful just by reading the internet. After pretraining, a model still needs to be steered toward answers people actually prefer. For years the standard recipe was reinforcement learning from human feedback, or RLHF, which works but is fiddly to run. In 2023 a Stanford team published a way to optimize the same goal with a much simpler training recipe. It is called Direct Preference Optimization, or DPO, and the paper's subtitle sums up the idea: "Your Language Model is Secretly a Reward Model."
This explainer covers what problem DPO solves, how it works in plain terms, what the paper actually measured, and where its limits are. You do not need to read math to follow it. If you want the formulas, they are in the original paper, in our technical DPO explainer, and in our DPO math guide.
The problem: pretrained models are hard to steer
A language model trained on a huge pile of text learns broad knowledge and some reasoning skill. But that text was written by people with many different goals and skill levels. The DPO authors give two examples. A coding assistant should understand common programming mistakes so it can fix them, yet when it writes code it should lean toward the high-quality code that exists in its training data. And a model should know that a common misconception exists without repeating it as true in half of its answers.
Picking the desired behavior out of everything the model can do is called steering or alignment. The usual way to do it is to collect human judgments about which of two answers is better, then train the model to produce more answers like the winners. If you want background on the wider topic of adapting a model after pretraining, see our explainer on what fine-tuning is.
How classic RLHF works
The paper describes the standard RLHF pipeline in three phases.
- Supervised fine-tuning (SFT). Start from a pretrained model and train it on high-quality examples of the task, such as dialogue or summaries.
- Reward modeling. Ask the SFT model for pairs of answers to the same prompt. Human labelers mark which answer they prefer. A separate reward model is then trained to score answers so that preferred ones get higher scores.
- Reinforcement learning. Use an RL algorithm, such as one called PPO, to push the language model toward answers the reward model scores highly. A penalty keeps the model from drifting too far from the SFT model, so it does not collapse into a few high-scoring but odd answers.
We cover this pipeline in more detail in our RLHF explainer and in our technical RLHF guide. The point for this article is the cost. The DPO paper says RLHF is "considerably more complex than supervised learning." It involves training multiple language models and sampling from the model during training, which adds compute cost. The authors also call RLHF "a complex and often unstable procedure."
The key idea behind DPO
DPO starts from the same goal as RLHF: maximize a reward while staying close to a reference model, which is usually the SFT model. The authors then use a piece of math that links rewards to model behavior. For this objective, the best possible model can be written in closed form in terms of the reward and the reference model. Turn that around, and the reward can be written in terms of the model and the reference model.
That flip is the whole trick. Instead of training a reward model and then running RL against it, you treat the language model itself as the reward model. The "reward" of an answer becomes beta times the log of how much more likely your model finds that answer than the reference model does. Beta is a setting that controls how closely the model stays tied to the reference. The paper calls this an implicit reward.
Once the reward is expressed that way, the preference-learning step turns into an ordinary classification problem. In plain terms, for each training example you have a prompt, a preferred answer and a rejected answer. The training loss pushes the model to raise the probability of the preferred answer relative to the reference model, and to lower the probability of the rejected one.
What the update does
The paper analyzes the gradient, which is the signal that tells the model how to change. It does two things at once: it increases the likelihood of the preferred answer and decreases the likelihood of the rejected one. It also weights each example by how wrong the model's implicit reward currently is. An example where the model already ranks the two answers correctly gets a small push. An example where it ranks them backwards gets a big push.
The authors say this weighting matters. They report that a naive version without it can cause the language model to degenerate. A related baseline they tested, called unlikelihood training, which simply pushes up the preferred answer and pushes down the rejected one without that constraint, produced generally meaningless responses on harder tasks such as summarization and dialogue.
What you need to run it
DPO needs three ingredients:
- A dataset of prompts, each with a preferred and a rejected answer.
- A reference model, normally the SFT model that the preference data was sampled from. If that model is not available, the paper says to build a reference by training on the preferred answers.
- The beta setting, which controls how strongly the model is held close to the reference. The paper uses beta of 0.1 by default and 0.5 for the summarization experiment.
The paper also notes that DPO does not need to sample from the language model during fine-tuning, and that it needs little hyperparameter tuning. The paper includes a short PyTorch snippet for the loss, which is a few lines long.
What the paper tested
The authors ran three tasks, all with models up to 6B parameters. They did not test anything larger, and they say so.
- Controlled sentiment generation. Starting from the start of a movie review in the IMDb dataset, the model had to write a positive continuation. The model was GPT-2-large, first fine-tuned on IMDb reviews. Because a sentiment classifier supplied the true reward, the authors could measure reward against how far each method drifted from the reference model. DPO gave the highest reward at every drift level they plotted, and it did so even compared with PPO given access to the true reward.
- Summarization. The task was to summarize Reddit posts from the TL;DR dataset. Judged by GPT-4 against human-written summaries, DPO won about 61% of comparisons at sampling temperature 0.0, against 57% for PPO at its best temperature, also 0.0.
- Single-turn dialogue. Using the Anthropic Helpful and Harmless dataset, the authors started from a Pythia-2.8B model. DPO was the only computationally efficient method that improved on the preferred answers in the dataset, and it matched or beat a costly baseline that samples 128 answers and keeps the best one.
Robustness and generalization
The paper reports two further results that explain why people paid attention to it.
First, DPO was less sensitive to the sampling temperature. PPO's summarization quality could fall to the level of the base GPT-J model at high temperatures, while DPO held up better.
Second, the authors tested the summarization models on a different kind of text, news articles from the CNN/DailyMail dataset, without retraining. GPT-4 win rates against the reference summaries were 0.36 for DPO and 0.26 for PPO at temperature 0, and 0.31 for DPO and 0.23 for PPO at temperature 0.25. The authors describe this as initial evidence that DPO generalizes about as well as PPO, not as a final answer.
Do humans agree with the GPT-4 judge?
Win rates from GPT-4 are only useful if they track what people think. The authors ran a human study on the summarization task. In it, DPO summaries sampled at temperature 0.25 were preferred 58% of the time over PPO summaries sampled at temperature 0. They also found that GPT-4 agreed with human raters about as often as human raters agreed with each other, which they take as support for using GPT-4 as a stand-in judge. They note that GPT-4 tended to prefer longer, more repetitive summaries than humans did when the prompt did not ask for concision, so the wording of the judging prompt matters. For more on why judge setup affects scores, see our guide to reading AI benchmark scores.
What the results do not show
It is easy to over-read a result like this. A few cautions come straight from the authors.
- Scale. The largest models tested were 6B parameters. The authors list scaling DPO to much larger models as future work.
- Out-of-distribution behavior. They call the generalization result initial and say a more comprehensive study is needed, including how a DPO model compares with one trained from an explicit reward function.
- Reward over-optimization. They ask how this problem shows up with DPO and whether a slight drop in performance late in one of their training curves is an example.
- The judge. GPT-4 win rates depend on the judging prompt, and the authors suggest future work on the best way to get high-quality judgments from automated systems.
- Failure cases are visible in the appendix. In one dialogue example, the DPO model gave a long, plausible answer about why the United States entered World War II that contained factually incorrect details, and GPT-4 preferred the dataset answer.
DPO also does not remove the need for preference data. Someone still has to collect, or generate, pairs of preferred and rejected answers, and the quality of those pairs sets the ceiling. What DPO removes is the separate reward model and the reinforcement learning loop.
How DPO relates to other techniques
DPO is one option in a larger set of post-training methods, not a replacement for all of them.
- Compared with RLHF using PPO, DPO optimizes the same underlying objective, which is reward maximization with a penalty for drifting from the reference. The paper argues DPO gets there more directly and more efficiently in the tasks it tested.
- Compared with supervised fine-tuning on good answers alone, DPO also uses the rejected answers, so the model learns what to avoid as well as what to imitate. The paper's "Preferred-FT" baseline, which trains only on preferred answers, did not improve much over the SFT model on summarization.
- Compared with parameter-efficient tuning such as LoRA, DPO answers a different question. LoRA is about how many weights you update. DPO is about what training signal you use. The paper does not test the two together.
What to take away
If you are a reader trying to understand how chat models get tuned, the short version is this. Preference tuning used to mean three stages, including a trained reward model and a reinforcement learning loop. DPO showed that, for the setups tested, you can fold the reward model into the language model and train with a classification-style loss instead. In the paper's experiments that matched or beat PPO-based RLHF on controlled sentiment, summarization and single-turn dialogue, with little tuning.
If you are a builder, the practical questions are about data and scale: whether you have good preference pairs, whether your reference model matches the data, and whether results from models up to 6B parameters carry over to the model you care about. The paper cannot answer those for you, and it says so. Before you pick a training library or setting, check that library's current documentation.
Sources
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv 2305.18290)
- What Is RLHF? Reinforcement Learning From Human Feedback Explained
- Reinforcement Learning from Human Feedback (RLHF): How Reward Models, PPO, and KL Penalties Align LLMs
- What Is Fine-Tuning? How AI Models Learn New Tasks
- What Is LoRA? Low-Rank Adaptation in Plain English
- What Are AI Benchmarks? How to Read AI Model Scores
- Direct Preference Optimization: How Implicit Reward Modeling Simplified LLM Alignment
- Direct Preference Optimization (DPO): Mathematical Foundations, Implicit Reward Derivation, Closed-Form Bradley-Terry Equivalence, and Reference Policy Regularization
