Group Relative Policy Optimization (GRPO): How Eliminating Value Models Scaled LLM Reasoning

Post-training reinforcement learning (RL) has become the primary mechanism for scaling reasoning capabilities in large language models. While early reinforcement learning from human feedback (RLHF) focused on conversational style and safety alignment, extending RL to multi-step reasoning domains such as mathematics, algorithmic coding, and formal logic exposed critical limitations in classical algorithms.

Standard Proximal Policy Optimization (PPO), long the foundational algorithm for instruction tuning, introduces massive compute overhead and optimization instability when applied to long reasoning chains. To overcome these constraints, DeepSeek introduced Group Relative Policy Optimization (GRPO) in DeepSeekMath and subsequently scaled the method in DeepSeek-R1 to train reasoning models without relying on traditional value networks.

Understanding how GRPO eliminates value estimation, normalizes relative rewards across sampled candidate groups, and enables autonomous self-correction is essential for modern post-training engineering.

The Computational Bottlenecks of Classical PPO

Classical Proximal Policy Optimization for LLMs, adapted from deep reinforcement learning and established in OpenAI's InstructGPT methodology, coordinates four separate models during training:

Actor Model ( $\pi_\theta$ ): The primary language model being optimized, generating tokens and updating parameters via policy gradients.
Reference Model ( $\pi_{ref}$ ): A frozen snapshot of the initial supervised fine-tuned (SFT) model, used to calculate Kullback-Leibler (KL) divergence and prevent policy collapse.
Critic / Value Model ( $V_\psi$ ): A learned neural network that estimates the expected cumulative future reward from any intermediate token state $s_t$ .
Reward Model ( $r_\phi$ ): A model trained on human preference pairs to assign scalar scores to completed responses.

Maintaining this four-model infrastructure imposes two severe bottlenecks during training:

1. Memory and Compute Overhead

The critic model typically shares the same parameter count and transformer architecture as the actor to maintain accurate state representations. In a distributed training setup, the critic requires its own forward passes, backward passes, optimizer states (such as AdamW first and second moments), and activation memory. For large models running across hundreds of GPU nodes, the critic effectively doubles memory consumption and consumes roughly half the post-training compute budget.

2. Value Function Drift on Long Contexts

In reasoning tasks, completions frequently extend from 4,000 to over 32,000 tokens as models explore proof steps, scratchpads, and code traces. Training a value network $V_\psi(s_t)$ to accurately predict the eventual success of a mathematical proof at token step 200 out of 10,000 is notoriously unstable. Value predictions suffer from extreme variance, noisy Generalized Advantage Estimation (GAE), and catastrophic capacity bottlenecks, frequently causing policy gradient updates to destabilize or collapse.

Core Mechanics of Group Relative Policy Optimization

Group Relative Policy Optimization resolves these bottlenecks by eliminating the critic model entirely. Instead of estimating a baseline with a learned neural network, GRPO calculates an empirical baseline directly from a group of sampled completions generated for the same prompt.

+-------------------------------------------------------------------+
|                           PROMPT (q)                              |
+-------------------------------------------------------------------+
                                  |
              Sample G independent completions from pi_theta
                                  |
    +-----------------+-----------------+-----------------+
    |                 |                 |                 |
    v                 v                 v                 v
[Completion 1]  [Completion 2]  [Completion 3]  [Completion G]
    |                 |                 |                 |
    v                 v                 v                 v
 [Score r_1]       [Score r_2]       [Score r_3]       [Score r_G]
    +-----------------+-----------------+-----------------+
                                  |
        Compute Group Mean: mu = (1/G) * sum(r_i)
        Compute Group Std:  sigma = sqrt((1/G) * sum((r_i - mu)^2))
                                  |
    +-------------------------------------------------------------+
    | Calculate Normalized Advantage: A_i = (r_i - mu) / (sigma)  |
    +-------------------------------------------------------------+
                                  |
      Apply Clipped Surrogate Policy Gradient with Token-Level KL

1. Group Sampling and Baseline Computation

For each training prompt $q$ sampled from dataset $P(Q)$ , GRPO generates a group of $G$ candidate outputs $\{o_1, o_2, \dots, o_G\}$ from the previous policy $\pi_{\theta_{old}}$ .

Each completion $o_i$ is evaluated by a reward mechanism (either a rule-based verifier or a reward model), yielding a scalar reward $r_i$ . GRPO computes the advantage $\hat{A}_i$ for completion $o_i$ by standardizing rewards across the sampled group:

$\mu = \frac{1}{G} \sum_{i=1}^G r_i$

$\sigma = \sqrt{\frac{1}{G} \sum_{i=1}^G (r_i - \mu)^2 + \epsilon}$

$\hat{A}_i = \frac{r_i - \mu}{\sigma}$

Where $\epsilon$ is a small constant (e.g., $10^{-8}$ ) to prevent division by zero. If a trajectory performs better than the group average, its advantage is positive ( $\hat{A}_i > 0$ ). If it performs worse, its advantage is negative ( $\hat{A}_i < 0$ ).

2. The GRPO Clipped Objective Function

The objective function optimizes the actor parameters $\theta$ across all $G$ trajectories while bounding the policy update magnitude using a PPO-style clipping mechanism:

$\mathcal{J}_{GRPO}(\theta) = \mathbb{E}_{q \sim P(Q), \{o_i\}_{i=1}^G \sim \pi_{\theta_{old}}} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left( \min \left( \rho_{i,t} \hat{A}_i, \text{clip}(\rho_{i,t}, 1-\epsilon, 1+\epsilon) \hat{A}_i \right) - \beta D_{KL}(\pi_\theta \parallel \pi_{ref}) \right) \right]$

Here, $\rho_{i,t}$ represents the token-level probability ratio:

$\rho_{i,t} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t} \mid q, o_{i,<t})}$

The objective applies the sequence-level normalized advantage $\hat{A}_i$ across every token $t$ in completion $o_i$ , scaled by the length of the completion $|o_i|$ .

3. Direct Token-Level KL Regularization

In standard PPO, KL divergence is often folded into the reward function as an adjusted reward term $r'_t = r_t - \beta D_{KL}$ . In GRPO, the KL divergence penalty is applied directly inside the loss objective.

To avoid negative or unstable KL estimates when sampling single sequences, DeepSeek employs an unbiased, non-negative estimator:

$D_{KL}(\pi_\theta \parallel \pi_{ref}) = \frac{\pi_{ref}(o_{i,t} \mid q, o_{i,<t})}{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} - \log \frac{\pi_{ref}(o_{i,t} \mid q, o_{i,<t})}{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} - 1$

This formulation penalizes deviations from the reference policy at each token step without requiring a separate value function to propagate discounted KL rewards backward through time.

Architectural Comparison: PPO vs. GRPO

The architectural differences between PPO and GRPO directly impact hardware efficiency, memory budgets, and post-training stability:

Active Models in Training: PPO requires 4 models (Actor, Critic, Reference, Reward). GRPO requires only 2 models (Actor and Reference). When combined with rule-based verifiers, the reward model is omitted as well.
VRAM Consumption: Eliminating the critic removes its optimizer states (such as FP32 Adam master weights and momentum buffers), cutting VRAM allocation per GPU by approximately 40% to 50%.
Backward Passes: PPO executes two backward passes per iteration (one for policy gradients on the actor, one for mean squared error on the critic). GRPO executes only one backward pass on the actor.
Credit Assignment Resolution: PPO computes token-level advantages via GAE based on local state values. GRPO assigns a sequence-level relative advantage broadcast across all tokens in the generated response.
Hyperparameter Sensitivity: PPO requires careful tuning of GAE decay ( $\lambda$ ), value loss coefficients ( $c_1$ ), and critic learning rates. GRPO removes value-related hyperparameters entirely, relying primarily on group size $G$ , clip ratio $\epsilon$ , and KL weight $\beta$ .

Verifiable Rule-Based Rewards in Reasoning Tasks

A critical driver of GRPO's effectiveness in mathematical and coding domains is the transition from neural reward models to deterministic, rule-based reward functions.

Neural reward models suffer from well-documented vulnerabilities:

Reward Hacking: The policy finds stylistic artifacts, formatting quirks, or repetitive loops that trigger high reward model scores without solving the underlying problem.
Length Bias: Reward models consistently favor longer, more verbose explanations, regardless of mathematical correctness.
Preference Drift: In multi-step formal derivations, slight intermediate errors are often overlooked by generalist neural reward classifiers.

Rule-Based Verification Pipeline

In technical domains, outputs can be verified programmatically:

Accuracy Rewards: Ground-truth mathematical verifiers (using symbolic engines like SymPy or exact string extraction) evaluate the final solution. In programming tasks, sandbox test execution validates code against unit tests. A binary reward ( $r_{acc} \in \{0, 1\}$ ) or fractional test pass rate is assigned.
Format and Syntax Rewards: Regular expressions enforce strict structuring tags, such as requiring reasoning steps inside <think>...</think> tags and the final answer inside <answer>...</answer> tags. Completions violating the formatting contract receive an immediate negative penalty ( $r_{fmt} = -1$ or $0$ ).

By combining accuracy and formatting rewards into a composite scalar $r_i = r_{acc} + r_{fmt}$ , the reinforcement learning system optimizes against ground truth without human labelers in the loop.

Emergence of Extended Thinking and Self-Correction

When trained with GRPO on verifiable reward domains, models exhibit emergent reasoning behaviors without explicit supervised demonstrations.

In DeepSeek-R1-Zero, where pure RL was applied directly to a base language model without prior SFT data, the model autonomously discovered complex reasoning strategies:

Dynamic Test-Time Computation: The average generation length expanded naturally as training progressed. The policy learned that generating detailed scratchpads, testing edge cases, and decomposing difficult questions increased the probability of finding the correct answer.
Autonomous Backtracking and Verification: Because group comparisons reward successful paths over unsuccessful ones, trajectories where the model recognized a calculation error, revised its approach, and corrected itself earned a higher relative advantage than trajectories that persisted in error.
Linguistic Re-evaluations: Phrases such as "Wait, let me double-check this step" or "This contradicts the earlier condition, let me restart" emerged spontaneously as optimal policy actions under relative group scoring.

Prompt: Solve for x: 3x + 5 = 20

Rollout 1: 3x = 15 => x = 5. Answer: 5               (Correct: r = 1.0)
Rollout 2: 3x = 25 => x = 8.33. Answer: 8.33         (Wrong:   r = 0.0)
Rollout 3: 3x = 15... Wait, 20-5=15 => x=5. Ans: 5   (Correct: r = 1.0)
Rollout 4: 3x = 20 + 5 = 25. Answer: 25              (Wrong:   r = 0.0)

Group Mean: 0.5 | Std: 0.5
Advantage (Rollouts 1 & 3): +1.0 (Gradient pushes model toward self-checking)
Advantage (Rollouts 2 & 4): -1.0 (Gradient penalizes flawed derivations)

Practical Challenges and Trade-Offs

Despite its memory efficiency and training stability, GRPO introduces distinct engineering challenges:

1. Intra-Group Variance Collapse

If all $G$ sampled outputs for a difficult prompt fail ( $r_i = 0$ for all $i$ ), the standard deviation is zero ( $\sigma = 0$ ). Even with epsilon smoothing, the resulting advantages are zero ( $\hat{A}_i = 0$ ), yielding zero gradient update for that prompt batch. Conversely, if all rollouts succeed ( $r_i = 1$ ), no learning occurs. GRPO requires careful prompt curation and dynamic sampling temperatures to ensure mixed success rates across groups.

2. Group Size Scaling

The statistical fidelity of the group baseline depends directly on group size $G$ . Setting $G=4$ or $G=8$ reduces generation latency but increases advantage variance. Setting $G=16$ or $G=64$ provides stable baselines but demands high concurrent generation throughput during rollout collection, shifting the computational bottleneck from backpropagation memory to inference serving capacity.

3. Length Drift and Repetition Loops

Without length-aware regularization, reasoning policies can develop reward hacking behaviors where tokens are padded unnecessarily to delay committing to a final answer. Implementing token budget penalties or progressive length clipping prevents uncontrolled context inflation.

Summary

Group Relative Policy Optimization represents a foundational shift in how language models are aligned for complex reasoning tasks. By replacing resource-intensive critic networks with empirical group statistics and pairing them with deterministic verification rewards, GRPO slashes training memory requirements while providing stable policy gradients across long-sequence derivations. As frontier models increasingly rely on test-time compute scaling and autonomous self-correction, group-relative policy algorithms provide the mathematical foundation for scalable post-training.