Group Relative Policy Optimization (GRPO): Mathematical Foundations, Group Baseline Advantage, Critic-Free Policy Gradients, and Reasoning Scaling

Reinforcement learning from human feedback (RLHF) and reinforcement learning with verifiable rewards (RLVR) have become central to post-training large language models. For years, the default policy optimization algorithm in LLM alignment was Proximal Policy Optimization (PPO). While PPO offers stable policy updates through clipped surrogate objectives and Generalized Advantage Estimation (GAE), it introduces severe computational and architectural overhead when scaled to hundred-billion-parameter foundation models.

Group Relative Policy Optimization (GRPO), introduced by Shao et al. in the DeepSeekMath framework and subsequently deployed to train DeepSeek-R1-Zero and DeepSeek-R1, modifies the actor-critic paradigm. By eliminating the separate critic (value) model and replacing token-level value estimation with a group-relative baseline computed across multiple candidate rollouts, GRPO reduces GPU memory footprints while stabilizing reinforcement learning for mathematical and logical reasoning tasks.

The Value Network Bottleneck in Standard PPO

In standard PPO applied to autoregressive language models, training typically requires four separate neural networks to reside in memory or coordinate across distributed ranks:

Policy Model (Actor, $\pi_\theta$ ): The active language model being trained.
Reference Model ( $\pi_{ref}$ ): A frozen copy of the supervised fine-tuned (SFT) model used to compute Kullback-Leibler (KL) divergence penalties.
Reward Model ( $r_\phi$ ): A model (or programmatic scoring function) that evaluates the quality of completions.
Value Model (Critic, $V_\psi$ ): A network, typically of identical parameter scale to the actor, initialized from the reward model and trained to predict the expected cumulative reward from each intermediate token state $s_t = (q, o_{<t})$ .

Maintaining a value network matching the size of a 70-billion or 671-billion parameter actor creates major engineering friction. First, the critic doubles the active parameter count that requires optimizer states (such as AdamW first and second moments), forward passes, and backward gradient computation. Second, predicting token-level value functions in generative language tasks is notoriously unstable: value prediction error frequently diverges during long chain-of-thought generations, propagating noisy advantage estimates through Generalized Advantage Estimation.

Mathematical Formulation of GRPO

GRPO resolves the critic bottleneck by removing the value network $V_\psi$ entirely. Instead of learning a parametric state-value estimator, GRPO estimates the baseline dynamically from a group of sampled responses to the same input query.

1. Group Sampling and Reward Assignment

For each prompt $q$ drawn from the training distribution $P(Q)$ , GRPO samples a group of $G$ distinct completions $\{o_1, o_2, \dots, o_G\}$ from the previous policy $\pi_{\theta_{old}}$ .

Each completion $o_i$ is evaluated by a scoring function to produce a scalar reward $r_i$ . This reward can be generated by neural preference models, deterministic rule-based verifiers, or a weighted combination of both:

$r_i = r_{verifiable}(q, o_i) + \alpha r_{format}(q, o_i)$

2. Group-Normalized Advantage Estimation

Rather than computing token-wise temporal difference errors $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ , GRPO computes a single advantage $\hat{A}_i$ for each response $o_i$ by standardizing the rewards across the group of $G$ candidates:

$\hat{A}_i = \frac{r_i - \text{mean}(\{r_1, r_2, \dots, r_G\})}{\text{std}(\{r_1, r_2, \dots, r_G\}) + \epsilon}$

Where:

$\text{mean}(\{r_1, \dots, r_G\}) = \frac{1}{G} \sum_{j=1}^G r_j$ acts as an empirical estimate of the expected reward $V(q)$ for prompt $q$ .
$\text{std}(\{r_1, \dots, r_G\}) = \sqrt{\frac{1}{G} \sum_{j=1}^G (r_j - \text{mean}(\mathbf{r}))^2}$ normalizes the advantage scale across easy and difficult prompts.
$\epsilon$ is a small numerical constant (typically $10^{-4}$ or $10^{-8}$ ) preventing division by zero when all outputs receive identical rewards.

Every token in completion $o_i$ is assigned the sequence-level advantage $\hat{A}_{i,t} = \hat{A}_i$ . This formulation directly reflects relative performance: completions scoring above the group average receive positive advantages ( $\hat{A}_i > 0$ ), reinforcing their token probabilities, while completions performing below the group average receive negative advantages ( $\hat{A}_i < 0$ ), suppressing their trajectories.

3. Clipped Surrogate Objective

The policy parameters $\theta$ are optimized by maximizing the GRPO surrogate objective:

$\mathcal{J}_{GRPO}(\theta) = \mathbb{E}_{q \sim P(Q), \{o_i\}_{i=1}^G \sim \pi_{\theta_{old}}(O|q)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left( \min \left( \rho_{i,t} \hat{A}_i, \; \text{clip}(\rho_{i,t}, 1-\varepsilon, 1+\varepsilon) \hat{A}_i \right) - \beta D_{KL}(\pi_\theta || \pi_{ref}) \right) \right]$

Where:

$\rho_{i,t} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t} \mid q, o_{i,<t})}$ represents the importance sampling probability ratio between the current policy and the rollout policy.
$\varepsilon$ is the PPO clipping parameter (typically set to 0.1 or 0.2), bounding the update to prevent destructive policy shifts.
$\beta$ is the regularization coefficient controlling drift from the reference model.
$|o_i|$ is the sequence length of completion $o_i$ , ensuring that long generations do not disproportionately dominate gradient updates relative to concise outputs.

4. Unbiased Analytical KL Divergence Approximation

In conventional RLHF implementations, the KL divergence penalty is often subtracted directly from token rewards inside the rollout phase. In GRPO, the KL divergence is integrated directly into the objective function. To calculate this penalty stably without introducing high sample variance, DeepSeekMath employs the non-negative unbiased estimator described by John Schulman:

$D_{KL}(\pi_\theta || \pi_{ref}) = \frac{\pi_{ref}(o_{i,t} \mid q, o_{i,<t})}{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} - \log \frac{\pi_{ref}(o_{i,t} \mid q, o_{i,<t})}{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} - 1$

Setting $k = \frac{\pi_{ref}}{\pi_\theta}$ , the function $f(k) = k - \log(k) - 1$ satisfies $f(1) = 0$ and $f(k) > 0$ for all $k \neq 1$ . This property guarantees that the penalty term is strictly positive whenever the active policy drifts from the reference policy, eliminating negative KL artifacts caused by finite-sample log-ratio approximations.

Verifiable Rule-Based Rewards and Reasoning Scaling

A critical property of GRPO is its synergy with rule-based reward systems in formal domains like mathematics, formal logic, and software engineering.

In traditional open-ended generation tasks, reward models must evaluate nuanced human preferences, which leaves them vulnerable to reward hacking (adversarial exploitation of reward model blind spots). In contrast, reasoning domains allow deterministic verification:

Accuracy Rewards: Binary scoring (1.0 for correct final answer, 0.0 for incorrect) extracted via regex parsers or validated with symbolic math engines (such as SymPy) and unit-test execution harnesses.
Format Rewards: Structural enforcement requiring the model to enclose its intermediate scratchpad within explicit markers, such as <think> and </think> tags, penalizing outputs that fail structural syntax constraints.

Emergence of Extended Reasoning Trajectories

When large models are trained via GRPO on verifiable tasks without intermediate supervised step-by-step demonstrations (as demonstrated in DeepSeek-R1-Zero), policy gradients driven by group-relative advantages induce emergent behaviors:

Autonomous Search Exploration: When faced with difficult problems, rollouts that generate exploratory reasoning paths eventually stumble upon correct solutions. Standardizing within the group assigns high positive advantages to these successful reasoning paths and negative advantages to incorrect short answers.
Self-Reflection and Error Correction: The model learns to generate reflective pivot phrases (such as "Wait, let me recalculate that" or "This leads to a contradiction, let us try an alternative substitution") without human demonstration data.
Dynamic Thinking Budgets: Generation lengths naturally expand as the policy allocates more computation to difficult prompts, effectively discovering test-time compute scaling through reinforcement learning exploration.

PPO vs. GRPO: Architectural and Computational Comparison

The structural differences between PPO and GRPO translate directly to training throughput and infrastructure requirements:

Memory Footprint: PPO requires memory for Actor weights, Critic weights, Actor optimizer states, Critic optimizer states, Reference model weights, and Reward model weights. GRPO requires memory only for Actor weights, Actor optimizer states, Reference model weights, and (optionally) Reward model weights. Eliminating the critic removes roughly 40% to 50% of the active VRAM allocated to trainable parameters and optimizer states.
Baseline Estimation: PPO relies on a learned parametric neural value network $V_\psi(s_t)$ , which suffers from value function approximation error and non-stationary targets. GRPO uses a non-parametric empirical mean $\frac{1}{G} \sum r_j$ across $G$ sampled completions for the prompt.
Advantage Computation: PPO uses Generalized Advantage Estimation (GAE) across temporal tokens with discount factor $\gamma$ and smoothing parameter $\lambda$ . GRPO applies sequence-level reward standardization $\frac{r_i - \mu}{\sigma + \epsilon}$ across the sampled group.
Sample Efficiency vs. Group Size: GRPO requires generating multiple completions per prompt ( $G \ge 4$ , with $G=16$ to $G=64$ common in practice). While generating $G$ completions increases forward-pass rollout computation, generation can be executed with highly optimized inference engines (e.g., vLLM or SGLang) with KV caching, while the heavy backpropagation pass operates over a streamlined single-model graph.

Engineering Considerations and Limitations

Deploying GRPO in production pipelines involves specific algorithmic trade-offs:

Reward Variance in Small Groups: If the group size $G$ is too small (e.g., $G=2$ ), the sample variance of the group standard deviation is high, leading to unstable advantage estimates. If all $G$ completions for a query fail or all succeed with identical scores, the numerator ( $r_i - \mu$ ) becomes zero, yielding zero gradient updates for that prompt.
Credit Assignment Granularity: Assigning a sequence-level advantage uniformly to all tokens in $o_i$ provides coarse credit assignment compared to fine-grained process-based reward models (PRMs). However, empirical results indicate that across sufficiently large training batches, policy gradient updates aggregate over token distributions to guide search behavior effectively.
Length Normalization: Dividing token losses by sequence length $|o_i|$ prevents gradient magnitude distortion across variable-length completions, preventing the optimization objective from favoring degenerate, repetitive output loops.

Group Relative Policy Optimization demonstrates that scaling reinforcement learning for reasoning does not require complex actor-critic infrastructure. By combining group-level baseline normalization with deterministic verifiers, GRPO provides a stable, memory-efficient framework for post-training LLM reasoning engines.