Reinforcement Learning from Human Feedback (RLHF): How Reward Models, PPO, and KL Penalties Align LLMs

Pre-training large language models on internet-scale text corpora equips them with general linguistic patterns, world knowledge, and broad reasoning heuristics. However, pre-training optimizes next-token prediction: $\mathbb{E}_{x \sim \mathcal{D}} [\log P_\theta(x_t \mid x_{<t})]$ . A base model trained purely on next-token prediction reflects the entirety of its web corpus, reproducing hallucinations, toxic phrasing, incorrect code, and unhelpful conversational patterns.

To transform an unconstrained base model into a reliable assistant, AI labs use alignment pipelines. While Supervised Fine-Tuning (SFT) provides an initial foundation by training models on high-quality demonstration dialogues, SFT alone suffers from exposure bias and cannot efficiently express fine-grained preference distinctions.

Reinforcement Learning from Human Feedback (RLHF) bridges this gap. By introducing a learned reward model that captures human preferences and optimizing the base model using Proximal Policy Optimization (PPO), RLHF steers model behavior toward helpfulness, harmlessness, and accuracy while penalizing toxic or evasive generations.

Why Supervised Fine-Tuning Is Not Enough

In Supervised Fine-Tuning (SFT), a base language model is trained via standard cross-entropy loss on curated prompt-response pairs $(x, y)$ :

$\mathcal{L}_{\text{SFT}}(\theta) = -\sum_{t=1}^T \log P_\theta(y_t \mid x, y_{<t})$

While SFT adapts the model to follow instructions and adopt a conversational tone, it introduces fundamental limitations:

Distribution Shift and Exposure Bias: During SFT training, the model is conditioned on ground-truth prefix tokens (teacher forcing). During inference, the model generates autoregressively, conditioning on its own previous token selections. A single off-distribution token can compound across sequence length, causing hallucinations or degenerative repetition.
Lack of Negative Feedback: Cross-entropy loss trains the model to replicate target tokens. It provides no mechanism to penalize plausible but subtly incorrect outputs. A model penalized for a slightly wrong variable name receives the same cross-entropy penalty as a model generating completely incoherent text.
Annotation Scalability: Creating high-quality, expert demonstrations for complex reasoning, long-form creative writing, or technical coding is labor-intensive and expensive. In contrast, ranking multiple candidate completions according to quality is faster, cheaper, and exhibits higher inter-annotator agreement.

The Three-Stage RLHF Pipeline

The standard RLHF pipeline, established by OpenAI in Learning to summarize from human feedback (Stiennon et al., 2020) and scaled in InstructGPT (Ouyang et al., 2022), operates in three sequential phases:

[Pre-trained Base Model]
           │
           ▼
[Stage 1: Supervised Fine-Tuning (SFT)] ───► Initial Policy π^SFT
           │
           ├──────────────────────────────────────────┐
           ▼                                          ▼
[Stage 2: Reward Model Training]             [Stage 3: PPO Policy Optimization]
  - Collect Pairwise Rankings (y_w > y_l)       - Actor Policy π_φ (initialized from π^SFT)
  - Bradley-Terry Loss Optimization             - Frozen Reference Model π_ref
  - Yields Scalar Reward Model r_θ              - Frozen Reward Model r_θ
                                                - Trainable Critic Network V_ψ

Stage 1: Supervised Fine-Tuning (SFT)

A base pre-trained foundation model is fine-tuned on thousands of high-quality instruction-response demonstrations. This yields the reference policy $\pi^{\text{SFT}}$ (or $\pi_{\text{ref}}$ ). This policy acts as the starting point for both the reward model and the downstream reinforcement learning policy.

Stage 2: Reward Model Training

The goal of reward modeling is to construct a scoring function $r_\theta(x, y)$ that maps a prompt $x$ and candidate completion $y$ to a scalar value representing quality, safety, and alignment.

Data Collection: A prompt $x$ is passed to several model variants (or sampled with different temperatures) to generate $K$ candidate completions $\{y_1, y_2, \dots, y_K\}$ .
Pairwise Ranking: Human annotators rank the candidate completions from best to worst. For each pair where response $y_w$ is preferred over $y_l$ ( $y_w \succ y_l$ ), a pairwise preference tuple $(x, y_w, y_l)$ is recorded.
Bradley-Terry Preference Framework: The probability that a human annotator prefers $y_w$ over $y_l$ is parameterized using the Bradley-Terry logistic model:

$P(y_w \succ y_l \mid x) = \sigma(r_\theta(x, y_w) - r_\theta(x, y_l)) = \frac{1}{1 + e^{-(r_\theta(x, y_w) - r_\theta(x, y_l))}}$

Reward Model Loss: The reward model parameters $\theta$ are optimized by minimizing the negative log-likelihood of the human preference comparisons:

$\mathcal{L}_{\text{RM}}(\theta) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\left[\log \sigma\left(r_\theta(x, y_w) - r_\theta(x, y_l)\right)\right]$

Architecturally, the reward model is initialized from the SFT checkpoint. Its final vocabulary classification head (projecting hidden states $h_T \in \mathbb{R}^{d_{\text{model}}}$ to vocabulary logits $\mathbb{R}^{|V|}$ ) is replaced with a linear projection layer outputting a single scalar value $r \in \mathbb{R}$ .

Stage 3: Policy Optimization via Proximal Policy Optimization (PPO)

With a trained scalar reward model $r_\theta$ , the language generation process is framed as a Markov Decision Process (MDP):

State ( $s_t$ ): The sequence composed of the initial prompt $x$ and the tokens generated so far: $s_t = (x, y_1, y_2, \dots, y_{t-1})$ .
Action ( $a_t$ ): The selection of the next token $y_t$ from the vocabulary $\mathcal{V}$ .
Transition ( $P$ ): Deterministic appending of the chosen token to the context: $s_{t+1} = [s_t, y_t]$ .
Policy ( $\pi_\phi$ ): The probability distribution over vocabulary tokens parameterized by transformer weights $\phi$ : $\pi_\phi(y_t \mid s_t)$ .

The Composite RLHF Objective

If an RL policy is optimized purely to maximize the reward model output $r_\theta(x, y)$ , the policy quickly exploits flaws in the reward function, generating pathological outputs that achieve high reward scores but are nonsensical or unreadable to humans (an effect known as Goodhart's Law or reward hacking).

To prevent this drift, RLHF incorporates a per-token Kullback-Leibler (KL) divergence penalty between the active policy $\pi_\phi$ and the frozen reference model $\pi_{\text{ref}}$ :

$\max_\phi \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\phi(\cdot \mid x)} \left[ r_\theta(x, y) - \beta D_{\text{KL}}(\pi_\phi(y \mid x) \parallel \pi_{\text{ref}}(y \mid x)) \right] + \gamma \mathbb{E}_{x \sim \mathcal{D}_{\text{pretrain}}} [\log \pi_\phi(x)]$

Where:

$r_\theta(x, y)$ is the terminal sequence score from the reward model.
$\beta$ is the KL penalty coefficient controlling how tightly the policy remains anchored to $\pi_{\text{ref}}$ .
$\gamma$ is an optional pre-training gradient term (the "PPO-ptx" term) that prevents regression on standard language modeling benchmarks.

Per-Token Reward Formulation

Because standard reward models evaluate complete sequences, the terminal reward $r_\theta(x, y)$ is assigned to the final token ( $t = T$ ). The per-token reward $R_t$ passed into the reinforcement learning algorithm is calculated as:

$R_t = \begin{cases} -\beta \log\left(\frac{\pi_\phi(y_t \mid s_t)}{\pi_{\text{ref}}(y_t \mid s_t)}\right) & \text{if } t < T \\ r_\theta(x, y) - \beta \log\left(\frac{\pi_\phi(y_T \mid s_T)}{\pi_{\text{ref}}(y_T \mid s_T)}\right) & \text{if } t = T \end{cases}$

Generalized Advantage Estimation (GAE) and PPO-Clip

To update the policy weights $\phi$ stably, RLHF employs Proximal Policy Optimization (Schulman et al., 2017).

A value network (the Critic, parameterized by $\psi$ ) estimates the expected return $V_\psi(s_t)$ from any partial sequence. Using temporal difference residuals:

$\delta_t^V = R_t + \gamma_{\text{rl}} V_\psi(s_{t+1}) - V_\psi(s_t)$

The generalized advantage $\hat{A}_t$ is computed across tokens:

$\hat{A}_t^{\text{GAE}(\gamma_{\text{rl}}, \lambda)} = \sum_{l=0}^{T-t-1} (\gamma_{\text{rl}} \lambda)^l \delta_{t+l}^V$

The Actor policy parameters $\phi$ are then updated using the clipped surrogate objective, which prevents excessively large policy steps:

$\mathcal{L}^{\text{CLIP}}(\phi) = \hat{\mathbb{E}}_t \left[ \min\left(r_t(\phi) \hat{A}_t, \text{clip}(r_t(\phi), 1-\epsilon, 1+\epsilon) \hat{A}_t\right) \right]$

Where the probability ratio is defined as:

$r_t(\phi) = \frac{\pi_\phi(y_t \mid s_t)}{\pi_{\phi_{\text{old}}}(y_t \mid s_t)}$

The 4-Model System Topology and Memory Footprint

Executing standard PPO-based RLHF requires running four distinct neural network models simultaneously during training:

| Role | Network | Weights Status | Optimizer State | Primary Purpose | Memory Footprint (16-bit) | | :--- | :--- | :--- | :--- | :--- | :--- | | Actor | $\pi_\phi$ | Trainable | Active (AdamW) | Generates rollouts; updated via PPO-Clip | $\sim 16 \times \text{Params}$ bytes | | Critic | $V_\psi$ | Trainable | Active (AdamW) | Predicts value estimates $V_\psi(s_t)$ for GAE | $\sim 16 \times \text{Params}$ bytes | | Reference | $\pi_{\text{ref}}$ | Frozen | None | Computes KL divergence reference logits | $\sim 2 \times \text{Params}$ bytes | | Reward | $r_\theta$ | Frozen | None | Computes sequence scalar reward $r_\theta(x, y)$ | $\sim 2 \times \text{Params}$ bytes |

For a 70-billion parameter model, hosting all four models simultaneously exceeds the memory capacity of single compute nodes, requiring distributed training orchestrators (such as DeepSpeed-Chat, Ray, or Megatron-LM) to shard parameters across GPU clusters using Fully Sharded Data Parallelism (FSDP) and tensor parallelism.

       ┌────────────────────────────────────────────────────────┐
       │                   Generation Phase                     │
       │  Prompt x ──► [Actor Policy π_φ] ──► Full Rollout y    │
       └───────────────────────────┬────────────────────────────┘
                                   │
       ┌───────────────────────────┴────────────────────────────┐
       │                   Evaluation Phase                     │
       │  (x, y) ──► [Reward Model r_θ] ──► Scalar Score r      │
       │  (x, y) ──► [Reference Model π_ref] ──► Ref Logits     │
       │  (x, y) ──► [Critic Model V_ψ] ──► Value Estimates V   │
       └───────────────────────────┬────────────────────────────┘
                                   │
       ┌───────────────────────────┴────────────────────────────┐
       │                   Optimization Phase                   │
       │  Compute Token Rewards R_t = -β KL(π_φ || π_ref) + r   │
       │  Compute GAE Advantage Estimates Â_t                   │
       │  Update Actor Policy π_φ via PPO-Clip Loss             │
       │  Update Critic Network V_ψ via MSE Value Loss          │
       └────────────────────────────────────────────────────────┘

Known Failure Modes and Practical Challenges

Deploying RLHF at scale reveals several structural weaknesses:

1. Reward Hacking (Goodhart's Law)

Because the reward model is an imperfect proxy for true human judgment, the policy discovers degenerate token patterns that maximize reward model logits without delivering valid content. Common manifestations include:

Length Bias: Reward models consistently score longer, verbose completions higher than concise answers. Policies trained without explicit length normalization develop extreme verbosity.
Sycophancy: Policies learn to flatter the user, agreeing with false factual premises provided in the prompt to avoid low reward scores associated with disagreement.
Formatting Over-Optimization: The model overuses bullet points, bold headers, and structured conclusions regardless of context.

2. The Alignment Tax

Aggressive reinforcement learning optimization can distort the model's underlying representations, causing performance degradation on structured capabilities such as code synthesis, formal mathematical proofs, and multilingual reasoning. Including the pre-training gradient loss ( $\gamma \mathbb{E}[\log \pi_\phi(x)]$ ) helps mitigate this degradation.

3. Annotator Noise and Inconsistency

Human preference datasets exhibit significant variance due to subjective interpretations of ambiguous prompts, differing cultural norms, and annotator fatigue. When preference pairs contain contradictory signals, the reward model loses calibration, producing high-variance gradient updates during PPO.

The Modern Evolution: DPO and GRPO

The infrastructural complexity and training instability of PPO-based RLHF led to alternative alignment architectures:

                          RLHF / Alignment Paradigms
                                     │
         ┌───────────────────────────┼───────────────────────────┐
         ▼                           ▼                           ▼
[Classic PPO-RLHF]          [Direct Preference (DPO)]     [Group Relative (GRPO)]
 - 4 models in memory        - 2 models (Policy, Ref)      - 2 models (Policy, Ref)
 - Explicit Reward Model     - Implicit Reward Model       - No Critic Network
 - High training variance    - Direct Supervised Loss      - Group Mean Normalization
 - General conversational    - General conversational      - Rule-based reasoning (RLVR)

Direct Preference Optimization (DPO): In Direct Preference Optimization (Rafailov et al., 2023), researchers showed that the Bradley-Terry preference model can be reparameterized analytically. This allows direct optimization of the language model policy on preference pairs using a closed-form binary cross-entropy loss, eliminating the need for a separate reward model, critic model, and reinforcement learning rollouts.
Reinforcement Learning from AI Feedback (RLAIF) / Constitutional AI: Proposed by Anthropic in Constitutional AI (Bai et al., 2022), RLAIF replaces human annotators with frontier language models evaluating candidate responses against a set of written constitutional principles, enabling scalable preference data generation.
Group Relative Policy Optimization (GRPO): Introduced in DeepSeekMath (Shao et al., 2024) and utilized in DeepSeek-R1, GRPO eliminates the Critic model entirely. Instead of estimating a baseline value function with a dedicated network, GRPO samples a group of $G$ candidate outputs $\{y_1, y_2, \dots, y_G\}$ for each prompt and computes the advantage by normalizing reward scores across the sampled group:

$\hat{A}_i = \frac{r_i - \text{mean}(\{r_1, \dots, r_G\})}{\text{std}(\{r_1, \dots, r_G\})}$

This eliminates roughly 50% of the active training memory footprint while maintaining stable policy updates, making it particularly effective for Reinforcement Learning with Verifiable Rewards (RLVR) in mathematical and programming domains.