Group Relative Policy Optimization (GRPO): Mathematical Foundations, Critic-Free Advantage Estimation, Group Reward Normalization, and Reinforcement Learning with Verifiable Rewards

Reinforcement learning from human and verifiable feedback has become the central paradigm for unlocking complex reasoning, mathematical problem solving, and autonomous code synthesis in frontier large language models. While early post-training pipelines relied heavily on Proximal Policy Optimization (PPO) or offline preference algorithms like Direct Preference Optimization (DPO), both methodologies present structural bottlenecks when scaling to long-horizon reasoning.

PPO requires maintaining a dedicated critic (value) network of comparable scale to the actor policy. This dual-network requirement doubles memory allocation and introduces instability during value function estimation across variable-length token sequences. Conversely, offline preference optimization methods like DPO cannot generate new reasoning trajectories on-policy, limiting their capacity for iterative self-discovery.

Group Relative Policy Optimization (GRPO), introduced by Shao et al. (2024) in DeepSeekMath and subsequently scaled in DeepSeek-R1 (Guo et al., 2025), resolves these architectural constraints. By eliminating the critic network entirely and estimating advantages directly from the relative scores of grouped sample rollouts, GRPO halves the memory overhead of online RL while establishing a robust framework for Reinforcement Learning with Verifiable Rewards (RLVR).

1. The Architectural Bottleneck of Standard PPO in LLMs

To understand the necessity of GRPO, one must examine the computational footprint of Proximal Policy Optimization (Schulman et al., 2017) when applied to autoregressive language models.

In standard PPO-based RLHF, the training infrastructure must instantiate four separate models in GPU memory simultaneously:

• Actor Policy ($\pi_\theta$): The active transformer undergoing optimization, requiring memory for parameters, gradients, and optimizer states (such as AdamW first and second moments).
• Critic / Value Network ($V_\phi$): A secondary transformer initialized from the reward or actor model that estimates the scalar expected return $V(s_t)$ for every intermediate token state $s_t$. It requires its own parameter, gradient, and optimizer allocations.
• Reference Policy ($\pi_{\text{ref}}$): A frozen checkpoint of the initial model used to compute Kullback-Leibler (KL) divergence penalties to prevent policy collapse.
• Reward Model ($r_\psi$): A frozen scoring model evaluating the quality of complete or partial trajectories.

Memory Footprint of the Value Model

In transformer training with 16-bit precision (FP16 or BF16) and 32-bit AdamW optimizer states, storing a model requires 16 bytes per parameter (2 bytes for weights, 2 bytes for gradients, 12 bytes for optimizer states). For a 70-billion-parameter actor, the model state alone consumes roughly 1.12 TB of VRAM across distributed nodes. Adding a 70B parameter critic network doubles this requirement, allocating another 1.12 TB strictly for value estimation before factoring in KV cache and activation memory.

Value Estimation Noise Across Token Sequences

Language generation is characterized by sparse rewards: a scalar score is typically awarded only at the final EOS token based on the correctness of the complete answer. PPO relies on Generalized Advantage Estimation (GAE) (Schulman et al., 2015) to propagate this terminal reward backward across all intermediate token positions:

$$\delta_t^V = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$$

$$\hat{A}_t^{\text{GAE}} = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l}^V$$

Fitting a token-level value function $V_\phi(s_t)$ on dense multi-thousand-token reasoning traces is notoriously difficult. Token-level value predictions suffer from high variance and regression drift, frequently destabilizing actor updates and requiring complex reward clipping and value loss hyperparameter tuning.

2. Mathematical Foundations of GRPO

GRPO bypasses the critic model entirely. Instead of approximating $V(s_t)$ with a neural network, GRPO samples a group of candidate outputs for each prompt, evaluates the group under a reward function, and calculates the advantage of each completion relative to the empirical mean and standard deviation of that specific group.

Step 1: Group Sampling

Given a prompt dataset distribution $P(Q)$, GRPO samples a batch of queries. For each individual query $q \sim P(Q)$, the policy generates a group of $G$ distinct candidate completions from the previous policy checkpoint $\pi_{\theta_{\text{old}}}$:

$$\mathcal{O}_q = \{o_1, o_2, \dots, o_G\} \sim \pi_{\theta_{\text{old}}}(O|q)$$

The generation is conducted with a non-zero temperature ($T > 0$) or top-$p$ nucleus sampling to ensure exploration across diverse reasoning trajectories.

Step 2: Group Relative Advantage Estimation

Each candidate completion $o_i$ is evaluated by a reward function $\mathcal{R}(q, o_i)$, yielding a scalar reward $r_i$:

$$\mathbf{r} = [r_1, r_2, \dots, r_G]$$

The baseline for the query $q$ is defined as the empirical sample mean of the group, and the scaling factor is the group sample standard deviation:

$$\bar{r} = \frac{1}{G} \sum_{i=1}^G r_i$$

$$\sigma_r = \sqrt{\frac{1}{G} \sum_{i=1}^G (r_i - \bar{r})^2}$$

The advantage $\hat{A}_i$ for completion $o_i$ is computed via standard score normalization:

$$\hat{A}_i = \frac{r_i - \bar{r}}{\sigma_r + \epsilon}$$

where $\epsilon > 0$ is a small constant (typically $10^{-4}$ or $10^{-6}$) preventing division by zero when all completions within the group achieve identical rewards.

In GRPO, the advantage $\hat{A}_i$ is a sequence-level scalar that reflects how much better or worse completion $o_i$ is compared to alternative paths generated for the exact same prompt. This advantage is broadcast across every generated token $t \in \{1, \dots, |o_i|\}$ in that sequence:

$$\hat{A}_{i,t} = \hat{A}_i \quad \forall t \in \{1, \dots, |o_i|\}$$

Step 3: Clipped Surrogate Policy Objective

Using the computed group advantages, GRPO optimizes the policy parameters $\theta$ using a clipped surrogate objective derived from PPO, penalized by a token-level KL divergence from the reference policy $\pi_{\text{ref}}$:

$$\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{q \sim P(Q), \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(O|q)} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left( \mathcal{L}_{i,t}^{\text{clip}}(\theta) - \beta D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})_{i,t} \right) \right]$$

The clipped surrogate loss $\mathcal{L}_{i,t}^{\text{clip}}(\theta)$ is defined as:

$$\mathcal{L}_{i,t}^{\text{clip}}(\theta) = \min \left( \frac{\pi_\theta(o_{i,t}|q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q, o_{i,<t})} \hat{A}_i, \; \text{clip}\left(\frac{\pi_\theta(o_{i,t}|q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q, o_{i,<t})}, 1 - \epsilon_{\text{clip}}, 1 + \epsilon_{\text{clip}}\right) \hat{A}_i \right)$$

where:

• The probability ratio $\rho_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t}|q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q, o_{i,<t})}$ measures the probability change under updated parameters $\theta$.
• $\epsilon_{\text{clip}}$ is the clipping threshold (typically set between $0.1$ and $0.2$).
• $\beta$ governs the strength of the KL divergence penalty.
• The objective normalizes the cumulative loss by the sequence length $|o_i|$ to ensure that longer reasoning outputs do not artificially dominate gradient updates over concise ones.

Step 4: Unbiased, Low-Variance KL Estimator

Rather than computing an expensive exact expectation over the full vocabulary $\mathcal{V}$ at every token position, GRPO adopts the unbiased, non-negative KL divergence estimator introduced by Schulman (2020):

$$D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})_{i,t} = \frac{\pi_{\text{ref}}(o_{i,t}|q, o_{i,<t})}{\pi_\theta(o_{i,t}|q, o_{i,<t})} - \log \left( \frac{\pi_{\text{ref}}(o_{i,t}|q, o_{i,<t})}{\pi_\theta(o_{i,t}|q, o_{i,<t})} \right) - 1$$

Defining the ratio $u = \frac{\pi_{\text{ref}}(o_{i,t}|q, o_{i,<t})}{\pi_\theta(o_{i,t}|q, o_{i,<t})}$, the function $f(u) = u - \log u - 1$ satisfies $f(1) = 0$ and $f(u) \ge 0$ for all $u > 0$ by the convexity of $-\log u$.

This estimator guarantees that the per-token regularization penalty is strictly non-negative at every step, avoiding the negative divergence artifacts that frequently arise when using the standard sample log-ratio estimator $\log \pi_\theta - \log \pi_{\text{ref}}$.

3. Structural Comparison: PPO vs. DPO vs. GRPO

To clarify how GRPO fits into the landscape of post-training algorithms, consider the operational mechanics of the leading paradigms:

Actor-Critic PPO

• Model Dependencies: Actor ($\theta$), Critic ($\phi$), Reference ($\pi_{\text{ref}}$), Reward ($r_\psi$).
• Advantage Mechanism: Temporal-difference GAE ($\hat{A}_t^{\text{GAE}}$) computed token-by-token using learned value function $V_\phi$.
• Training Mode: Online, iterative on-policy rollout generation.
• Memory Footprint: High (dual parameter, gradient, and optimizer state allocations for actor and critic).
• Primary Weakness: Value network training instability, memory constraints limiting batch size and context window length.

Direct Preference Optimization (DPO)

• Model Dependencies: Actor ($\theta$), Reference ($\pi_{\text{ref}}$).
• Advantage Mechanism: Implicit reward derived mathematically from closed-form Bradley-Terry log-ratio margin $(r(x,y_w) - r(x,y_l))$.
• Training Mode: Offline on static datasets of pre-collected winner/loser pairs $(x, y_w, y_l)$.
• Memory Footprint: Low (single active network).
• Primary Weakness: Cannot explore novel reasoning trajectories outside the static training corpus; susceptible to out-of-distribution degradation and likelihood displacement.

Group Relative Policy Optimization (GRPO)

• Model Dependencies: Actor ($\theta$), Reference ($\pi_{\text{ref}}$), Verifiable Rule Verifier or Reward Model.
• Advantage Mechanism: Empirical standardization across $G$ sampled completions per prompt ($\hat{A}_i = (r_i - \bar{r})/\sigma_r$).
• Training Mode: Online, iterative on-policy rollout generation.
• Memory Footprint: Low (critic network eliminated, freeing ~50% VRAM for larger batch sizes and extended context lengths).
• Primary Advantage: Full on-policy exploration for multi-step reasoning while maintaining the lightweight memory footprint of offline methods.

4. Statistical Properties and Variance Dynamics

The theoretical properties of GRPO have been analyzed through the lens of classical U-statistics (Xi et al., 2026), demonstrating how group-level advantage estimation balances sample efficiency and gradient variance.

Group Size ($G$) Scaling

The hyperparameter $G$ defines the sample size drawn for each query during rollout. Selecting an appropriate $G$ involves direct trade-offs:

• Small Group Size ($G = 2$ to $4$): Low compute overhead during rollout generation. However, sample variance of $\bar{r}$ and $\sigma_r$ is high. When $G=2$, if one output succeeds ($r=1$) and one fails ($r=0$), the advantages simplify to $+1$ and $-1$. If both outputs share the same score, the advantage collapses to zero.
• Optimal Group Size ($G = 8$ to $16$): Provides a robust empirical estimate of the prompt difficulty distribution. In DeepSeekMath and DeepSeek-R1, $G$ is typically configured between $8$ and $16$, striking a balance between rollout GPU time and policy gradient variance reduction.
• Large Group Size ($G \ge 32$): Diminishing returns in variance reduction relative to the linear increase in inference generation FLOPs.

Automatic Filtering of Trivial and Impossible Prompts

A critical mathematical property of group standardization is its self-regulating gradient behavior:

• All-Correct Groups: If a query is trivial and all $G$ completions succeed ($r_i = 1.0 \; \forall i$), the sample variance $\sigma_r = 0$. The numerator $(r_i - \bar{r}) = 0$, yielding $\hat{A}_i = 0$. The objective produces zero gradient for this prompt, preventing the model from overfitting on already mastered tasks.
• All-Incorrect Groups: If a query is too difficult and all $G$ completions fail ($r_i = 0.0 \; \forall i$), $\sigma_r = 0$ and $\hat{A}_i = 0$. The prompt contributes zero gradient updates, protecting the policy from destabilizing updates driven by ungrounded negative feedback.
• Mixed Success Groups: Gradients are activated exclusively on prompts where the model exhibits variance ($0 < \sum r_i < G$). The policy gradient selectively reinforces the specific search branches and reasoning steps that differentiated the winning completions from the failing ones.

5. Reinforcement Learning with Verifiable Rewards (RLVR)

The most transformative application of GRPO is in Reinforcement Learning with Verifiable Rewards (RLVR). In domains governed by formal rules (such as mathematics, competitive programming, and formal logic), neural reward models can be replaced entirely with deterministic rule-based evaluators.

Reward Function Decomposition

In RLVR pipelines, the scalar reward $r_i$ is typically constructed from two complementary components:

$$r_i = r_{\text{accuracy}}(q, o_i) + \alpha \cdot r_{\text{format}}(o_i)$$

1. Accuracy Reward ($r_{\text{accuracy}}$): A deterministic verifier evaluating final answer correctness:

• For mathematical reasoning: Symbolic equivalence verification using computer algebra systems (e.g., SymPy) comparing extracted answers against ground truth (reward $1.0$ for correct, $0.0$ for incorrect).
• For code synthesis: Sandboxed execution against unit test suites (reward based on percentage of passed tests or all-or-nothing binary execution).

1. Format Reward ($r_{\text{format}}$): Structural enforcement requiring the model to partition its reasoning process:

• Requiring the generation to encapsulate thinking traces within explicit XML tags, such as <think> ... </think> followed by the final answer.
• Assigning a penalty or zero reward if the model omits tags, fails to close tags, or leaks unstructured scratchpad tokens into the final response.

Emergence of Complex Reasoning Traces

When trained via GRPO on verifiable mathematical problems without supervised chain-of-thought demonstrations, models exhibit spontaneous behavioral phase shifts during training (Guo et al., 2025):

• Autonomous CoT Length Expansion: The average sequence length generated during thinking grows organically as the policy discovers that spending additional compute tokens on intermediate steps improves its probability of landing on the correct verifiable solution.
• Self-Reflection and Verification: The model learns to generate explicit self-checking markers (such as "Wait, let me recalculate that" or "Let me double check this step"), re-evaluating earlier intermediate conclusions and backtracking when an inconsistency is detected.
• Search and Backtracking: The policy navigates branching problem-solving paths, systematically discarding unpromising sub-goals in favor of alternative algebraic strategies.

In benchmark evaluations on DeepSeekMath 7B (Shao et al., 2024), applying GRPO over base instruction tuning improved accuracy on GSM8K from 82.9% to 88.2% and on the rigorous competition-level MATH benchmark from 46.8% to 51.7%, matching or surpassing significantly larger proprietary baselines.

6. Implementation and Infrastructure Considerations

Deploying GRPO at scale requires orchestrating distributed inference and training loops efficiently:

• Decoupled Rollout and Training Engines: Rollout generation demands high-throughput, memory-efficient inference serving engines (such as vLLM or SGLang) leveraging PagedAttention and continuous batching. Once trajectories $\mathcal{O}_q$ are generated, activations and tokens are passed via shared memory or NCCL to training workers running Fully Sharded Data Parallelism (FSDP) or Megatron-LM ZeRO-3.
• Loss Masking on Input Prompts: The loss calculation in GRPO must strictly mask out prompt tokens $q$. Gradients and KL penalties are evaluated exclusively over the completion tokens $t \in \{1, \dots, |o_i|\}$.
• Sequence Packing: Because reasoning traces vary widely in token length, naive padding wastes substantial tensor core throughput. Modern implementations pack variable-length rollouts into continuous 1D token tensors, utilizing FlashAttention varlen kernels to eliminate padding overhead.
• Reference Model Caching: To avoid redundant forward passes through the reference model $\pi_{\text{ref}}$ during backpropagation, reference log-probabilities $\log \pi_{\text{ref}}(o_{i,t}|q, o_{i,<t})$ can be precomputed and cached during the initial rollout collection stage.

GRPO represents an essential simplification in post-training reinforcement learning: by substituting a complex, resource-heavy value function with statistical group comparison, it unlocks efficient on-policy learning and scalable reasoning across frontier models.

Sources

• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (Shao et al., 2024)
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (Guo et al., 2025)
• Proximal Policy Optimization Algorithms (Schulman et al., 2017)
• High-Dimensional Continuous Control Using Generalized Advantage Estimation (Schulman et al., 2015)
• Approximating KL Divergence (Schulman, 2020)
• Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic (Xi et al., 2026)
• Direct Preference Optimization: Your Language Model Is Secretly a Reward Model (Rafailov et al., 2023)