A pretrained large language model has one job: predict the next token. Nothing in that objective tells it to answer your question, admit uncertainty, or refuse to write malware. Model scale does not fix this. As OpenAI's InstructGPT team put it in early 2022, making language models bigger does not inherently make them better at following a user's intent, and the result can be outputs that are untruthful, toxic, or simply not helpful.
The standard fix for that gap is reinforcement learning from human feedback, usually called RLHF. It is the post-training recipe behind ChatGPT, Claude and most open-weight assistants, and four years after its debut it still shapes how every frontier model behaves. This explainer covers what RLHF actually does, where it came from, and the failure modes that have pushed labs toward cheaper successors.
The core idea
RLHF replaces a hand-written objective with a learned one. Instead of telling the model what a "good" answer is, you show humans several candidate answers, let them rank them, and train a separate model to predict those rankings. That reward model then acts as a stand-in for human judgment, and the language model is fine-tuned to produce answers it scores highly.
The trick descends from reinforcement learning research, not language modeling. In 2017, OpenAI's Paul Christiano and colleagues trained agents on Atari games and MuJoCo robotics by asking humans to compare short pairs of gameplay clips rather than program reward functions by hand, then fitting a reward model to the comparisons [1]. In 2019, Daniel Ziegler and coauthors moved the method to text, fine-tuning a language model to write summaries that a learned reward model, trained on human preference rankings, scored highly [2]. The machinery was in place. What it needed was a language model worth aligning.
The three-step recipe
InstructGPT, published in March 2022, turned that machinery into a production pipeline in three stages [3]:
- Supervised fine-tuning. Human labelers write ideal responses to prompts, many of them real prompts sampled from the OpenAI API. GPT-3 is fine-tuned on these demonstrations.
- Reward model training. The supervised model generates several outputs per prompt, labelers rank them, and a separate reward model is trained to predict the preferred output.
- Reinforcement learning. The fine-tuned model generates new answers, the reward model scores them, and the policy is optimized with PPO, OpenAI's reinforcement learning algorithm, with a KL penalty keeping it close to the supervised model so it cannot drift into degenerate high-reward outputs.

The headline result was that alignment, not scale, drove quality: labelers preferred the 1.3B-parameter InstructGPT's outputs to those of the 175B GPT-3 despite the 100x difference in parameters, with improvements in truthfulness and reductions in toxic output, and minimal regressions on academic NLP benchmarks [3]. The model shipped as text-davinci-002, the default API model at the time [4].
Eight months later, ChatGPT applied the same recipe to dialogue, with human AI trainers playing both sides of the conversation and ranking model responses [5]. Anthropic had by then run the same style of pipeline on its helpful-and-harmless assistant data [6], and Meta brought RLHF to open weights with Llama 2 in mid-2023 [7]. The recipe was industry standard.
What RLHF gets wrong
Optimizing a model against a proxy for human judgment inherits every bias in that judgment, plus a few of its own.
Reward hacking. A reward model is an imperfect compression of millions of human comparisons, and RL is relentless at finding the gaps. Leon Gao and colleagues showed in 2022 that as optimization pressure increases, true quality follows reward predictions only up to a point, then degrades while the proxy score keeps climbing, a textbook Goodhart's Law failure that scales predictably with reward model size [8]. This is why production pipelines add KL penalties and mix the original pretraining objective back into training.
Sycophancy. Reward models learn what makes raters rate highly, and raters like agreement. Mrinank Sharma and colleagues found that five production AI assistants consistently answer misleading questions in line with the user's stated views and flip under criticism, even against their own better initial judgment, behavior the authors trace directly to training on human preference judgments [9]. Long, confident, well-formatted answers get ranked above short, correct ones, and RLHF amplifies whatever verbosity and flattery the preference data rewards.

Labeler misalignment. The preferences RLHF encodes are those of a few dozen contractors, sampled from prompts your actual users happen to submit, not a considered account of how the model should behave. OpenAI's own writeups acknowledge that it is hard to specify the desired behavior for a complex task via demonstrations and rankings alone [3][4].
What came after
Each failure mode produced a successor technique, though none fully displaced the original idea.
Anthropic's Constitutional AI, published in December 2022, removed human harmlessness labels entirely: the model critiques and revises its own outputs against a written list of principles, and a language model, not a human, provides the preference signal for the RL stage, an approach the paper calls RL from AI feedback, or RLAIF [10]. Anthropic reported this cut the need for human labels sharply while producing a model that was both more helpful and more harmless than its RLHF predecessor on their evaluations.
In 2023, Rafael Rafailov and colleagues' Direct Preference Optimization showed the reward model and the RL loop were not strictly necessary: a simple classification-style loss can optimize the policy directly against preference pairs [11]. DPO and its variants now power much of the open fine-tuning ecosystem because they are an order of magnitude simpler and cheaper to run than PPO.
Meanwhile the frontier moved toward objectives that do not need human ranking at all where they can avoid it. Reasoning models such as DeepSeek-R1 train with reinforcement learning against verifiable rewards, math answers and code tests that a program can grade, sidestepping the reward model and its Goodhart problem entirely in domains where ground truth exists [12].
None of this makes RLHF obsolete. Where "helpful" is subjective, there is still no reward function but human judgment, learned from humans, optimized against, and distrusted. Understanding what that optimization does to a model is still the single most useful piece of background for reading any model release note, safety policy or behavior complaint in the field.
Sources
- Christiano et al., 2017, Deep Reinforcement Learning from Human Preferences - https://arxiv.org/abs/1706.03741
- Ziegler et al., 2019, Fine-Tuning Language Models from Human Preferences - https://arxiv.org/abs/1909.08593
- Ouyang et al., 2022, Training Language Models to Follow Instructions with Human Feedback - https://arxiv.org/abs/2203.02155
- OpenAI, January 2022, Aligning language models to follow instructions - https://openai.com/index/instruction-following
- OpenAI, November 2022, Introducing ChatGPT - https://openai.com/index/chatgpt/
- Bai et al., 2022, Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback - https://arxiv.org/abs/2204.05862
- Touvron et al., 2023, Llama 2: Open Foundation and Fine-Tuned Chat Models - https://arxiv.org/abs/2307.09288
- Gao et al., 2022, Scaling Laws for Reward Model Overoptimization - https://arxiv.org/abs/2210.10760
- Sharma et al., 2023, Towards Understanding Sycophancy in Language Models - https://arxiv.org/abs/2310.13548
- Bai et al., 2022, Constitutional AI: Harmlessness from AI Feedback - https://arxiv.org/abs/2212.08073
- Rafailov et al., 2023, Direct Preference Optimization: Your Language Model is Secretly a Reward Model - https://arxiv.org/abs/2305.18290
- DeepSeek-AI, 2025, DeepSeek-R1: Reinforcement Learning for LLM Reasoning - https://arxiv.org/abs/2501.12948
