Ask a chatbot a question and it answers in a helpful, polite format. That behavior did not come for free. A model trained only to predict the next word on internet text can produce answers that are, in the words of OpenAI's InstructGPT paper, "untruthful, toxic, or simply not helpful." The step that turned raw language models into assistants people could use is called RLHF: reinforcement learning from human feedback.
This explainer walks through what RLHF is, where it came from, how the three-step recipe works, what it fixed, what it did not fix, and the newer methods that try to do the same job more simply. Every claim below comes from the original papers or the lab posts linked in the text.
The problem RLHF solves
The 2022 InstructGPT paper from OpenAI states the problem in one line: "Making language models bigger does not inherently make them better at following a user's intent." The authors note that large models can produce output that is untruthful, toxic, or simply not helpful. In their words, these models are "not aligned with their users."
The root cause is the training goal. Pretraining teaches a model to guess the next word in a huge pile of text. That makes it fluent and knowledgeable, but "sounds like the internet" is not the same goal as "gives this person a good answer." You need some way to tell the model what a good answer looks like.
The obvious fix is to write a scoring rule. But most of what people want from an answer (clear, correct, on topic, not preachy, not rude) is hard to write down as a formula. People can, however, look at two answers and say which one is better. RLHF is a way to turn those simple judgments into a training signal.
Where the idea came from
The core idea predates chatbots. In 2017, researchers published Deep reinforcement learning from human preferences. They trained agents to play Atari games and control simulated robots without giving them the game's reward function. Instead, people watched pairs of short video clips of the agent and picked the one that looked closer to the goal.
The paper reports that this worked with feedback on less than one percent of the agent's interactions with its environment, and that it could teach complex new behaviors with about an hour of human time. The key point for later work: you do not need people to rate everything. You need enough comparisons to train a model of what people prefer, and then that model does the rating.
In 2020, the same approach was applied to language in Learning to summarize from human feedback. The team collected a large dataset of human comparisons between summaries of Reddit posts, trained a model to predict which summary people would prefer, and used it as a reward to fine-tune a summarizer. According to the paper, the resulting models significantly outperformed both the human-written reference summaries and much larger models trained with supervised learning alone. The authors also found that optimizing their learned reward model produced better summaries, as judged by people, than optimizing ROUGE, the standard automatic summary score.
The three-step recipe
The version most people mean by "RLHF" is the recipe in the InstructGPT paper. It starts from a pretrained GPT-3 model and runs three steps.
Step 1: Show the model good answers
The team gathered prompts, some written by labelers and some submitted through the OpenAI API. Labelers then wrote demonstrations of the desired behavior: the answer a helpful assistant should give. The model was fine-tuned on these examples with ordinary supervised learning. This step is often called supervised fine-tuning, or SFT.
SFT alone gets you part of the way. The model learns the format and tone of a good answer.
Step 2: Teach a second model to judge
Next, the team had the model produce several outputs for each prompt and asked labelers to rank them.
Those rankings train a reward model: a separate network that reads a prompt and an answer and outputs a score for how much a person would likely prefer it. In effect, the reward model is a learned stand-in for the human raters.
Step 3: Practice against the judge
Finally, the fine-tuned model is trained with reinforcement learning. It writes answers, the reward model scores them, and the model is updated to produce higher-scoring answers. OpenAI used an algorithm called Proximal Policy Optimization (PPO) for this step, as its ChatGPT launch post also describes.
There is one important guardrail. As the DPO paper summarizes the standard setup, RLHF maximizes the estimated reward "without drifting too far from the original model." Without that limit, the model can learn to exploit quirks in the reward model and produce text that scores well but reads badly.
What RLHF achieved
The headline result from InstructGPT is about size. In human evaluations on the team's prompt distribution, outputs from the 1.3 billion parameter InstructGPT model were preferred to outputs from the 175 billion parameter GPT-3, even though it had 100 times fewer parameters. The paper also reports improvements in truthfulness, fewer toxic outputs, and only minimal performance drops on public NLP benchmarks.
In other words, a small amount of targeted human feedback mattered more for usefulness than a huge increase in model size.
Later in 2022, OpenAI used the same approach for ChatGPT. Its launch post says the model was trained "using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup." Human AI trainers wrote example conversations in which they played both the user and the assistant. To build the reward model, trainers ranked several alternative completions of model-written messages. The model was then fine-tuned with PPO, and OpenAI says it ran several iterations of this process.
What RLHF did not fix
The same sources are candid about the limits.
- Mistakes remain. The InstructGPT abstract says the model "still makes simple mistakes."
- No source of truth. OpenAI's ChatGPT post says the model sometimes writes plausible-sounding but incorrect answers, and that fixing this is hard partly because "during RL training, there's currently no source of truth."
- Raters have biases, and the model learns them. The same post says ChatGPT was often too verbose and overused certain phrases. OpenAI attributes this partly to trainers preferring longer answers that look more comprehensive, and partly to "well-known over-optimization issues." It cites a paper titled Scaling Laws for Reward Model Overoptimization on that second point.
- Caution has a cost. The post notes that training the model to be more cautious causes it to decline questions it could answer correctly.
The general lesson: RLHF optimizes for what the reward model thinks people prefer. That is close to "good," but not identical. Where the two differ, the model will follow the reward model.
Simpler alternatives: DPO and RLAIF
RLHF works, but the full pipeline is heavy. You train and run a separate reward model, then run a reinforcement learning loop that samples many answers from the model during training. Two later lines of research try to keep the benefits while cutting the cost.
DPO: skip the reward model
In 2023, researchers published Direct Preference Optimization: Your Language Model is Secretly a Reward Model. The paper calls RLHF "a complex and often unstable procedure." Its insight is mathematical: with a different way of writing down the reward, the best model for a given set of preferences can be computed in closed form. That turns the whole problem into a simple classification loss on the human comparison data.
The paper says DPO is stable and lightweight, with no need to sample from the model during fine-tuning or to do significant hyperparameter tuning. In its experiments, DPO beat PPO-based RLHF at controlling the sentiment of generated text, and matched or improved response quality on summarization and single-turn dialogue, while being "substantially simpler to implement and train."
DPO still needs human preference data. What it removes is the separate reward model and the reinforcement learning loop.
For the math behind it, see our technical DPO explainer.
RLAIF: let an AI do the rating
The 2022 paper Constitutional AI: Harmlessness from AI Feedback attacks a different cost: human labels. For training harmlessness, the only human oversight in the method is a written list of rules or principles, which the authors call a constitution.
The process has two phases. In the supervised phase, the model writes answers, critiques its own answers against the principles, revises them, and is fine-tuned on the revisions. In the reinforcement learning phase, a model compares pairs of answers to decide which is better, and those AI judgments train the preference model used as the reward. The authors call this "RL from AI Feedback," or RLAIF. The paper reports that the result was a harmless but non-evasive assistant that engages with harmful questions by explaining its objections, and that the methods control AI behavior "with far fewer human labels."
Our technical Constitutional AI explainer covers the critique and revision loop in more depth.
Why it matters
RLHF is the bridge between a model that knows a lot and an assistant that is useful. The InstructGPT result showed that the bridge can matter more than raw scale, and the ChatGPT launch showed what that looks like in a product.
For anyone reading new AI research, the vocabulary from this family shows up constantly. "SFT" is the first step of the recipe. A "reward model" or "preference model" is the learned judge. "PPO" is the reinforcement learning algorithm OpenAI used. "DPO" means the same preference data was used without a separate reward model. "RLAIF" means AI judgments replaced some or all of the human ones.
It also helps explain familiar chatbot habits. If an assistant is wordy, over-cautious, or confidently wrong, part of the reason is that it was trained to please a learned model of human raters. The labs that built these systems say as much in their own write-ups.
Want the math? See our technical guide to how RLHF works.
Sources
- Training language models to follow instructions with human feedback (InstructGPT), arXiv 2203.02155
- Deep reinforcement learning from human preferences, arXiv 1706.03741
- Learning to summarize from human feedback, arXiv 2009.01325
- Introducing ChatGPT, OpenAI
- Scaling Laws for Reward Model Overoptimization, arXiv 2210.10760
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model, arXiv 2305.18290
- Constitutional AI: Harmlessness from AI Feedback, arXiv 2212.08073
