Subscribe
Education
5 min read

Rejection Sampling Fine-Tuning: How Filtering Model Outputs by Reward Replaced RLHF Complexity

RAFT, short for reward ranked fine-tuning (Dong et al., 2023), samples several completions per prompt, ranks them with a reward model and fine-tunes on the best one. The three-step loop, typical settings, the paper's reported results and how it differs from best-of-N at inference.

Rejection Sampling Fine-Tuning: How Filtering Model Outputs by Reward Replaced RLHF Complexity
Rejection Sampling Fine-Tuning

Rejection sampling fine-tuning replaces PPO's policy-gradient loop with a simpler one: generate several completions per prompt, score them with a reward model, keep the best, and fine-tune on the filtered data. Dong et al. (2023) formalized the loop as RAFT (Reward rAnked FineTuning), and Llama 2 used rejection sampling fine-tuning alongside PPO (Touvron et al., 2023). Each round turns the reinforcement learning step into ordinary supervised fine-tuning on the selected samples.

From Computational Statistics to LLM Alignment

The name originates in statistics: when you cannot sample directly from a target distribution, you sample from an easier proposal distribution and accept or reject each sample based on a weighting function. In LLMs, the target distribution is "high-quality completions," the proposal distribution is the current model, and the acceptance criterion is a reward model score.

WebGPT (Nakano et al., 2021), Anthropic's Helpful and Harmless work (Bai et al., 2022), and OpenAI's process reward models (Lightman et al., 2023) all used variants. The method was formalized as RAFT (Reward rAnked FineTuning) by Dong et al. (2023) and independently deployed at scale in Llama 2 (Touvron et al., 2023).

The Three-Step Iteration

At each iteration, RAFT executes:

  1. Data collection: Sample K completions per prompt from the current policy using temperature-controlled sampling (typically 0.7 to 1.0, with K = 10 to 30). This balances diversity against quality.
  2. Reward-based ranking: Pass every (prompt, completion) pair through a trained reward model R. Select the highest-scoring completion per prompt (argmax per row) or the top K pairs globally.
  3. Supervised fine-tuning: Run standard SFT on the selected pairs using the same cross-entropy loss as instruction tuning.

Mathematically, given prompts X = [x₁, ..., xₘ] and N completions per prompt forming matrix Y, the reward model produces matrix R where rᵢⱼ = R(yᵢⱼ | xᵢ). The per-prompt selection function S(R) = [argmaxⱼ r₁ⱼ, ..., argmaxⱼ rₘⱼ] yields chosen completions Y_chosen. The model then minimizes −Σ log π_θ(y_chosen | x).

RAFT three-step loop: generate K completions, reward model selects best, SFT on chosen

Llama 2 applied rejection sampling only to its 70B model and fine-tuned the smaller models on that model's filtered samples, which the Llama 2 paper describes as distilling the larger model into the smaller ones. Dong et al. test a similar distillation route (see Extensions below).

Why It Works: Stability Over Sophistication

PPO-based RLHF requires four models in memory simultaneously (policy, reference, critic, reward), careful KL scheduling, and gradient updates that can destabilize. RAFT needs only one model loaded at a time. In Dong et al.'s learning curves, reward rises steadily across RAFT iterations.

Dong et al. also report that perplexity stayed stable across RAFT training, while the perplexity of the PPO agent usually got worse quickly as reward increased.

On LLaMA-7B with the HH-RLHF dataset, the RAFT-aligned model reached the highest mean reward in Dong et al.'s table and beat PPO baselines in GPT-4 and human evaluations on 100 test prompts. In a separate diffusion experiment, restoring Stable Diffusion 1.5's ability to generate 256×256 images, RAFT reached scores similar to DDPO with about 50 times less compute: 8.4 minutes versus 415 minutes of training on a single A40 GPU (the paper's Table 9).

Best-of-N Inference vs. Training-Time Filtering

Best-of-N (BoN) uses the same generate-and-score pipeline at inference time: sample N completions, return the highest-reward one. RAFT instead bakes the selection into the weights. BoN spends compute per query; RAFT spends compute once during training. They are complementary: BoN can further boost a RAFT-aligned model at test time.

For the general technique beyond RAFT, including the math of best-of-N selection, iterative rounds, RSO and common failure modes, see our guide to rejection sampling fine-tuning and best-of-N.

Practical Hyperparameters

The RLHF Book (Lambert, 2025) documents settings that work in practice:

  • Temperature: 0.7 to 1.0 (higher early in training, lower later)
  • Completions per prompt: 10 to 30 or more (fewer makes selection noisy; more yields diminishing returns)
  • Selection strategy: Argmax per prompt is standard; top-K overall can concentrate on "easy" prompts
  • Deduplication: Often applied before scoring to avoid wasting reward model calls on near-duplicate outputs
  • Iterative scheduling: Re-generate from the updated checkpoint each round; Llama 2 accumulated top samples across all prior iterations

Extensions and Variants

KL regularization: Add a penalty β · KL(π_θ || π_ref) to the SFT loss to prevent catastrophic drift from the base model's capabilities.

Knowledge distillation: The decoupled generation/training design lets a small student learn from a large teacher's filtered outputs. Dong et al. showed GPT-Neo-2.7B improving significantly when trained on RAFT-aligned LLaMA-7B samples.

Statistical Rejection Sampling Optimization (RSO): Liu et al. (2024) provide a principled analysis connecting rejection sampling to DPO and IPO objectives, showing it implicitly optimizes a variant of the pairwise preference loss.

Cross-Modal Generality

RAFT's only modality-specific component is the reward model. The same loop aligns diffusion models: generate K images per prompt, score with a text-image alignment reward (e.g., CLIP or a trained aesthetic predictor), fine-tune on the best. This generality makes it a default alignment tool for any generative foundation model.

When to Use It

  • You have a reward model (or verifiable reward function for reasoning tasks)
  • You want alignment without PPO's infrastructure burden
  • You need to align models larger than your training GPU memory allows for multi-model RLHF
  • You want interpretable, inspectable training data: the filtered dataset is the alignment signal

Sources

EducationLLMAlignmentpost-training