
Rejection sampling fine-tuning replaces PPO's policy-gradient loop with a simpler one: generate several completions per prompt, score them with a reward model, keep the best, and fine-tune on the filtered data. Dong et al. (2023) formalized the loop as RAFT (Reward rAnked FineTuning), and Llama 2 used rejection sampling fine-tuning alongside PPO (Touvron et al., 2023). Each round turns the reinforcement learning step into ordinary supervised fine-tuning on the selected samples.
From Computational Statistics to LLM Alignment
The name originates in statistics: when you cannot sample directly from a target distribution, you sample from an easier proposal distribution and accept or reject each sample based on a weighting function. In LLMs, the target distribution is "high-quality completions," the proposal distribution is the current model, and the acceptance criterion is a reward model score.
WebGPT (Nakano et al., 2021), Anthropic's Helpful and Harmless work (Bai et al., 2022), and OpenAI's process reward models (Lightman et al., 2023) all used variants. The method was formalized as RAFT (Reward rAnked FineTuning) by Dong et al. (2023) and independently deployed at scale in Llama 2 (Touvron et al., 2023).
The Three-Step Iteration
At each iteration, RAFT executes:
- Data collection: Sample K completions per prompt from the current policy using temperature-controlled sampling (typically 0.7 to 1.0, with K = 10 to 30). This balances diversity against quality.
- Reward-based ranking: Pass every (prompt, completion) pair through a trained reward model R. Select the highest-scoring completion per prompt (argmax per row) or the top K pairs globally.
- Supervised fine-tuning: Run standard SFT on the selected pairs using the same cross-entropy loss as instruction tuning.
Mathematically, given prompts X = [x₁, ..., xₘ] and N completions per prompt forming matrix Y, the reward model produces matrix R where rᵢⱼ = R(yᵢⱼ | xᵢ). The per-prompt selection function S(R) = [argmaxⱼ r₁ⱼ, ..., argmaxⱼ rₘⱼ] yields chosen completions Y_chosen. The model then minimizes −Σ log π_θ(y_chosen | x).

Llama 2 applied rejection sampling only to its 70B model and fine-tuned the smaller models on that model's filtered samples, which the Llama 2 paper describes as distilling the larger model into the smaller ones. Dong et al. test a similar distillation route (see Extensions below).
Why It Works: Stability Over Sophistication
PPO-based RLHF requires four models in memory simultaneously (policy, reference, critic, reward), careful KL scheduling, and gradient updates that can destabilize. RAFT needs only one model loaded at a time. In Dong et al.'s learning curves, reward rises steadily across RAFT iterations.
Dong et al. also report that perplexity stayed stable across RAFT training, while the perplexity of the PPO agent usually got worse quickly as reward increased.
On LLaMA-7B with the HH-RLHF dataset, the RAFT-aligned model reached the highest mean reward in Dong et al.'s table and beat PPO baselines in GPT-4 and human evaluations on 100 test prompts. In a separate diffusion experiment, restoring Stable Diffusion 1.5's ability to generate 256×256 images, RAFT reached scores similar to DDPO with about 50 times less compute: 8.4 minutes versus 415 minutes of training on a single A40 GPU (the paper's Table 9).
Best-of-N Inference vs. Training-Time Filtering
Best-of-N (BoN) uses the same generate-and-score pipeline at inference time: sample N completions, return the highest-reward one. RAFT instead bakes the selection into the weights. BoN spends compute per query; RAFT spends compute once during training. They are complementary: BoN can further boost a RAFT-aligned model at test time.
For the general technique beyond RAFT, including the math of best-of-N selection, iterative rounds, RSO and common failure modes, see our guide to rejection sampling fine-tuning and best-of-N.
Practical Hyperparameters
The RLHF Book (Lambert, 2025) documents settings that work in practice:
- Temperature: 0.7 to 1.0 (higher early in training, lower later)
- Completions per prompt: 10 to 30 or more (fewer makes selection noisy; more yields diminishing returns)
- Selection strategy: Argmax per prompt is standard; top-K overall can concentrate on "easy" prompts
- Deduplication: Often applied before scoring to avoid wasting reward model calls on near-duplicate outputs
- Iterative scheduling: Re-generate from the updated checkpoint each round; Llama 2 accumulated top samples across all prior iterations
Extensions and Variants
KL regularization: Add a penalty β · KL(π_θ || π_ref) to the SFT loss to prevent catastrophic drift from the base model's capabilities.
Knowledge distillation: The decoupled generation/training design lets a small student learn from a large teacher's filtered outputs. Dong et al. showed GPT-Neo-2.7B improving significantly when trained on RAFT-aligned LLaMA-7B samples.
Statistical Rejection Sampling Optimization (RSO): Liu et al. (2024) provide a principled analysis connecting rejection sampling to DPO and IPO objectives, showing it implicitly optimizes a variant of the pairwise preference loss.
Cross-Modal Generality
RAFT's only modality-specific component is the reward model. The same loop aligns diffusion models: generate K images per prompt, score with a text-image alignment reward (e.g., CLIP or a trained aesthetic predictor), fine-tune on the best. This generality makes it a default alignment tool for any generative foundation model.
When to Use It
- You have a reward model (or verifiable reward function for reasoning tasks)
- You want alignment without PPO's infrastructure burden
- You need to align models larger than your training GPU memory allows for multi-model RLHF
- You want interpretable, inspectable training data: the filtered dataset is the alignment signal
Sources
- Dong et al., "RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment," arXiv:2304.06767 (2023), published in TMLR. https://arxiv.org/abs/2304.06767
- Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models," arXiv:2307.09288 (2023). https://arxiv.org/abs/2307.09288
- Nakano et al., "WebGPT: Browser-Assisted Question-Answering with Human Feedback," arXiv:2112.09332 (2021). https://arxiv.org/abs/2112.09332
- Bai et al., "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback," arXiv:2204.05862 (2022). https://arxiv.org/abs/2204.05862
- Lightman et al., "Let's Verify Step by Step," ICLR 2024, arXiv:2305.20050. https://arxiv.org/abs/2305.20050
- Lambert, "Rejection Sampling," Chapter 9 in Reinforcement Learning from Human Feedback, 2025. https://rlhfbook.com/c/09-rejection-sampling
- Liu et al., "Statistical Rejection Sampling Improves Preference Optimization," ICLR 2024. https://arxiv.org/abs/2309.06657
- Yuan et al., "Scaling Relationship on Learning Mathematical Reasoning with Large Language Models," arXiv:2308.01825 (2023), which introduced the name Rejection sampling Fine-Tuning (RFT). https://arxiv.org/abs/2308.01825
