Group Relative Policy Optimization (GRPO): Mathematical Foundations, Group-Relative Baselines, and Critic-Free Reinforcement Learning in Reasoning Models

Reinforcement learning from human and automated feedback has become the central mechanism for eliciting complex reasoning, tool use, and mathematical problem-solving in large language models. For years, Proximal Policy Optimization (PPO) served as the default algorithmic framework for on-policy alignment. However, standard PPO introduces significant infrastructure overhead when applied to multi-billion parameter models. PPO requires instantiating and training a separate value network (the critic) alongside the policy model (the actor), roughly doubling the GPU memory required for training parameters, optimizer states, and forward-backward activations.

To resolve this systems bottleneck while optimizing policy exploration in complex reasoning domains, DeepSeek researchers introduced Group Relative Policy Optimization (GRPO) in the DeepSeekMath foundation model. GRPO discards the parameterized critic network entirely. Instead, it computes advantage estimates dynamically by generating a group of candidate outputs for each prompt, evaluating their task rewards, and standardizing the resulting scores against the empirical mean and standard deviation of the group.

This architectural shift reduces training-time memory consumption, eliminates the instability of value function approximation, and aligns naturally with rule-based outcome verification in mathematical reasoning and code generation.

The Infrastructure and Optimization Bottlenecks of Actor-Critic PPO

To understand why critic-free reinforcement learning is essential for scaling reasoning models, one must examine the computational footprint of standard actor-critic PPO in large language models.

In classical PPO applied to autoregressive sequence generation, the training harness typically orchestrates four distinct models or model states:

1. The Policy Model ($\pi_\theta$): The active actor network undergoing optimization via policy gradients.
2. The Reference Model ($\pi_{\text{ref}}$): A frozen snapshot of the initial supervised fine-tuned (SFT) policy, used to compute Kullback-Leibler (KL) divergence penalties to prevent policy collapse.
3. The Reward Model ($r_\psi$): A neural preference model (or automated scoring harness) that provides scalar evaluations of completed generations.
4. The Value Network ($V_\phi$): The critic network, parameterized identically or similarly to the policy, trained via mean squared error regression to predict the expected discounted cumulative return from every token state.

The Token-Level Advantage and GAE Recursion

Standard PPO uses the value network to compute Generalized Advantage Estimation (GAE) across the generated sequence of length $T$. At each token step $t \in \{1, \dots, T\}$, the temporal difference (TD) residual $\delta_t$ is computed as:

$$\delta_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$$

The generalized advantage estimate $\hat{A}_t^{\text{GAE}(\gamma, \lambda)}$ is then derived recursively:

$$\hat{A}_t^{\text{GAE}(\gamma, \lambda)} = \sum_{l=0}^{T - t - 1} (\gamma \lambda)^l \delta_{t+l}$$

While GAE provides an effective bias-variance trade-off for token-level credit assignment, it imposes severe systems penalties:

• Memory Duplication: The critic network $V_\phi$ possesses an architecture comparable in size to the actor $\pi_\theta$. In a 70-billion parameter training run, maintaining the critic adds 140 GB of raw weights in FP16/BF16, plus 560 GB of FP32 optimizer states (first and second moments under AdamW), along with full activation memory during backward passes.
• Value Approximation Error: In long-form reasoning trajectories (such as chain-of-thought derivations exceeding thousands of tokens), intermediate token states often carry ambiguous value. When the reward signal is sparse and only provided at the final answer token, fitting a dense token-level value function $V_\phi(s_t)$ produces high variance and noisy gradient updates that can destabilize the policy.

Mathematical Formulation of GRPO

Group Relative Policy Optimization sidesteps the need for a parameterized value network by substituting a statistical group baseline in place of $V_\phi(s)$.

1. Group Rollout Sampling

For each training prompt or question $q$ drawn from the training distribution $P(Q)$, GRPO samples a cohort of $G$ independent candidate outputs $\{o_1, o_2, \dots, o_G\}$ using the current rollout policy $\pi_{\theta_{\text{old}}}$:

$$o_i \sim \pi_{\theta_{\text{old}}}(O \mid q), \quad \text{for } i \in \{1, 2, \dots, G\}$$

In production implementations, $G$ typically ranges from 8 to 64 outputs per query, depending on distributed memory constraints and batch topologies.

2. Task Reward Computation and Group Normalization

Each sampled completion $o_i$ is evaluated by a scoring function to produce a scalar reward $r_i$. In verifiable domains such as mathematics and programming, $r_i$ is typically computed by deterministic rule-based verifiers:

• Correctness Reward ($r_{\text{acc}}$): Binary indicator (e.g., $+1.0$ for matching the ground-truth final mathematical expression, $0.0$ otherwise) or unit test pass rate.
• Formatting Reward ($r_{\text{format}}$): Structural penalty or bonus enforcing that reasoning steps are correctly enclosed within structured tags (e.g., <think> ... </think>).

Given the set of scalar rewards $\{r_1, r_2, \dots, r_G\}$ for prompt $q$, GRPO computes the empirical sample mean and sample standard deviation:

$$\bar{r} = \frac{1}{G} \sum_{i=1}^G r_i$$

$$\sigma_r = \sqrt{\frac{1}{G} \sum_{i=1}^G (r_i - \bar{r})^2 + \epsilon}$$

where $\epsilon > 0$ is a small numerical stabilizer (e.g., $10^{-8}$) preventing division by zero when all outputs in a group receive identical rewards.

The group-relative advantage $\hat{A}_i$ for candidate output $o_i$ is defined as the standardized z-score:

$$\hat{A}_i = \frac{r_i - \bar{r}}{\sigma_r}$$

Crucially, $\hat{A}_i$ is a sequence-level advantage applied uniformly across all tokens in completion $o_i$. Completions that outperform the group average receive positive advantage ($\hat{A}_i > 0$), while completions that score below the group average receive negative advantage ($\hat{A}_i < 0$).

3. The Objective Function and Surrogate Loss

The objective function optimized by GRPO incorporates PPO-style probability ratio clipping with a per-token length normalization and an unbiased sample-level KL divergence penalty:

$$J_{\text{GRPO}}(\theta) = \mathbb{E}_{\substack{q \sim P(Q) \\ \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}}} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \mathcal{L}_{i,t}(\theta) \right]$$

The per-token clipped surrogate objective $\mathcal{L}_{i,t}(\theta)$ is defined as:

$$\mathcal{L}_{i,t}(\theta) = \min \left( \rho_{i,t}(\theta) \hat{A}_i, \; \text{clip}(\rho_{i,t}(\theta), 1 - \epsilon_{\text{clip}}, 1 + \epsilon_{\text{clip}}) \hat{A}_i \right) - \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})_{i,t}$$

where the probability ratio $\rho_{i,t}(\theta)$ is:

$$\rho_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})}$$

and $\epsilon_{\text{clip}}$ is the clipping threshold (typically set to $0.2$).

The Non-Negative Unbiased KL Divergence Estimator

In classical reinforcement learning implementations, the KL divergence penalty between the optimized policy $\pi_\theta$ and the reference model $\pi_{\text{ref}}$ is frequently incorporated directly into the reward function:

$$r_{\text{total}}(s_t, a_t) = r(s_t, a_t) - \beta \log \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\text{ref}}(a_t \mid s_t)}$$

However, when optimizing multi-turn sequence models without a value network, incorporating the KL term into the reward alters the scale of $r_i$ and distorts the empirical standard deviation $\sigma_r$ during group standardization.

Instead, GRPO optimizes the KL divergence directly within the surrogate loss function using an unbiased, non-negative sample estimator. Specifically, for each token $o_{i,t}$, the divergence is calculated as:

$$D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})_{i,t} = \frac{\pi_{\text{ref}}(o_{i,t} \mid q, o_{i,<t})}{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} - \log \left( \frac{\pi_{\text{ref}}(o_{i,t} \mid q, o_{i,<t})}{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} \right) - 1$$

Mathematical Derivation of Non-Negativity

Let $u = \frac{\pi_{\text{ref}}(o_{i,t} \mid \cdot)}{\pi_\theta(o_{i,t} \mid \cdot)}$ denote the likelihood ratio of the reference model to the active policy. The estimator function is:

$$f(u) = u - \log(u) - 1$$

1. At $u = 1$ (where the policy and reference distributions match exactly):

$$f(1) = 1 - \log(1) - 1 = 0$$

1. Taking the first derivative with respect to $u$:

$$f'(u) = 1 - \frac{1}{u}$$

1. For $u > 1$, $f'(u) > 0$ (monotonically increasing).
2. For $0 < u < 1$, $f'(u) < 0$ (monotonically decreasing).

Because $f(u)$ attains its global minimum at $u = 1$ where $f(1) = 0$, the estimator $D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})_{i,t} \ge 0$ holds strictly for all possible token probabilities. This formulation eliminates negative KL divergence artifacts that arise from naive log-ratio sampling estimators, ensuring stable regularization throughout training.

Architectural Comparison: PPO vs. GRPO

The structural differences between PPO and GRPO translate into substantial shifts in computational efficiency, memory consumption, and training dynamics:

• Critic Model Requirement: PPO requires a full value network $V_\phi$ of equivalent parameter scale to the actor policy. GRPO eliminates the critic network entirely, reducing parameter count and backward graph overhead.
• Baseline Estimation Mechanism: PPO relies on parameterized state-value predictions $V_\phi(s_t)$ across time steps. GRPO derives its baseline directly from the empirical group mean reward across sampled completions $\bar{r} = \frac{1}{G}\sum_{i=1}^G r_i$.
• Advantage Computation: PPO calculates Generalized Advantage Estimation (GAE) at every individual token step. GRPO uses a standardized z-score advantage applied uniformly across the completion tokens.
• Model Copies in Accelerator Memory: PPO typically requires keeping four distinct model instances (Policy, Reference, Reward, and Critic) in GPU cluster memory. GRPO operates with two or three (Policy, Reference, and optional rule-based Verifier).
• Optimizer Memory Overhead: PPO must maintain AdamW moment states for both the actor and critic (consuming 8x parameter size in FP32). GRPO only tracks optimizer states for the actor policy (consuming 4x parameter size in FP32).
• Cluster VRAM Savings: Removing the critic network and its optimizer states yields a 40% to 50% net reduction in total GPU memory requirements during training.
• Exploration Topologies: PPO relies on individual trajectory updates against a learned state value. GRPO evaluates a group cohort simultaneously, providing comparative rank-based credit across diverse reasoning paths.

Why GRPO Excels in Long-Form Reasoning Models

The widespread adoption of GRPO in models such as DeepSeek-R1 stems from several mathematical and structural synergies between group-relative baselines and chain-of-thought exploration:

1. Eliminating Value Network Bottlenecks in Long Sequences

In reasoning tasks where generation lengths reach 8,000 to 32,000 tokens of self-reflection and backtracking, training a value network to accurately predict intermediate step values across millions of tokens becomes computationally intractable. GRPO evaluates the full trajectory outcome, avoiding the need to fit an unstable intermediate value landscape.

2. Natural Self-Consistency Exploration

By sampling $G$ diverse outputs per question, GRPO mirrors the mechanics of self-consistency decoding during the training rollout phase. The model naturally explores alternative proof paths, arithmetic decompositions, and code structures. Trajectories that arrive at the verified solution are rewarded proportionally against those that fail within the same prompt context.

3. Length Normalization Prevents Verbosity Exploitation

In standard policy gradients, sequence-level rewards without length normalization introduce a positive bias toward overly long sequences. Longer sequences accumulate more token-level policy gradient updates for a positive advantage:

$$\sum_{t=1}^{|o_i|} \nabla_\theta \log \pi_\theta(o_{i,t}) \hat{A}_i$$

GRPO applies an explicit $\frac{1}{|o_i|}$ scalar factor across the token sum for each completion. This normalizes the gradient contribution across completions of varying lengths, ensuring that the model does not artificially inflate token count purely to amplify reward gradients.

Implementation Considerations and Edge Cases

Deploying GRPO in production training clusters requires addressing several practical edge cases:

Handling Zero-Variance Groups ($\sigma_r = 0$)

When a prompt is either trivial (all $G$ rollouts pass verification, yielding $r_i = 1.0, \forall i$) or excessively difficult (all $G$ rollouts fail, yielding $r_i = 0.0, \forall i$), the sample standard deviation evaluates to $\sigma_r = 0$.

In these scenarios, the numerator $r_i - \bar{r} = 0$. The advantage evaluates to $\hat{A}_i = 0$ for all completions in the group. Consequently, the policy gradient contribution for this prompt is zero, effectively masking out non-informative queries from gradient updates. This automatic filtering ensures that training updates concentrate computational capacity on prompts at the model's active learning frontier.

Group Size Scaling ($G$)

The choice of group size $G$ governs the trade-off between baseline estimator variance and rollout throughput:

• Small $G$ ($G \in [2, 4]$): High variance in $\bar{r}$ and $\sigma_r$, leading to noisy advantage estimates and potential training instability.
• Moderate to Large $G$ ($G \in [8, 64]$): Provides a robust empirical estimate of the outcome distribution. In large-scale reasoning runs, setting $G=16$ or $G=64$ amortizes prompt prefill computation across multiple parallel decode workers, maximizing GPU arithmetic utilization.

Sources

• Shao, Z., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300.
• Guo, D., et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948.
• Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
• Schulman, J., et al. (2016). High-Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv:1506.02438.