A long-running AI agent does its work in turns. It writes a search query, reads the results, calls a tool, reads the output, and eventually gives a final answer. Each of those turns is a block of text that the model writes one token at a time. When the agent is trained with reinforcement learning (RL), it usually gets one score at the very end: the task passed or it failed. The hard part is deciding which turns, and which tokens inside them, deserve credit or blame for that score.
A new paper, Multiscale Policy Optimization for Long-Horizon LLM Agents, argues that most current RL recipes pick the wrong unit for this. Its method, MPO, assigns credit mainly per turn and then adjusts it per token. The authors are from Peking University, IQuest Research, Tsinghua University, Renmin University of China, The University of Hong Kong and one independent researcher. The paper was posted to arXiv on October 26, 2026.
The problem: two kinds of RL, two wrong scales
The paper sorts today's methods into two groups.
- Critic-free, group-based methods such as GRPO, DAPO and GSPO compare several attempts at the same task and give each whole attempt one shared score. Every turn and every token in a rollout gets the same credit.
- Critic-based, token-level methods such as PPO, VC-PPO, VAPO and SAO train a second network, the critic, to predict the expected outcome at each token. They treat every token as a separate action. That gives finer credit, but it does not separate "the model is still writing this turn" from "the model just acted on the environment."
The authors say agent training needs both: credit for each complete interaction, and a way to tell the tokens inside that interaction apart.
The evidence: critic values jump between turns
The starting point is an observation about the critic itself. The team looked at 2,048 trajectories and 16,323 turns from eight checkpoint batches of one search-agent training run. Differences between turn averages explain a median of 89.5% of the critic's value variance within a trajectory. Between-turn variation was larger than within-turn variation in 93.8% of trajectories.
In plain terms, the critic's estimate of "how well is this going" stays fairly flat while the model writes a turn, then moves when the environment answers. That suggests the turn, not the token, is the natural place to pass credit back through time.
The paper also notes the other side (Appendix C). Small value changes inside turns add up over many tokens, and boundaries account for only 9.2% of the pooled squared step-to-step value changes. So token detail still matters. It just should not drive the long-range credit.
How MPO works
MPO has two parts.
1. Multiscale Advantage Estimation (MAE). First, MAE runs generalized advantage estimation (GAE, the standard way PPO turns critic values into credit) on a time axis of turns instead of tokens. Credit decays once per interaction, not once per token, so the length of a turn does not change how far credit travels back. Second, MAE computes a short local trace inside each turn and centers it, so the token-level adjustments always average out to the turn's credit. A single knob, beta, sets how strong the local adjustment is. With beta set to 0, every token in a turn gets the same credit.
The two decay settings differ on purpose. With the paper's defaults, the turn trace reaches back about five turns, while the token trace reaches about twenty tokens.
2. Turn-level importance correction. RL methods like PPO limit how far each update moves the model by comparing new and old probabilities. MPO computes that ratio for the whole turn, not for each token. A stop-gradient trick keeps each token's own credit in the gradient while the full-turn ratio controls the weighting and clipping. The paper gives a theoretical bound showing that, under the same KL limits, token-level ratios carry an extra penalty that the turn-level version avoids.
Results
The team trained Qwen3-4B-Thinking and Qwen3-30B-A3B-Instruct on three settings: search, Claw-Eval and Automation Bench. Baselines were GRPO, GSPO, PPO, VAPO and SAO.
In Table 1, MPO has the highest reported score on all nine metrics at both model sizes. Against the strongest non-MPO baseline in each column, the pass@1 gains are:
- 4B: BrowseComp-Plus +4.33 points (48.08 vs 43.75 for SAO), GAIA +3.85 (39.47 vs 35.62 for SAO), HLE +0.66 (9.92 vs 9.26 for GSPO).
- 30B-A3B: BrowseComp-Plus +3.59 (50.32 vs 46.73 for VAPO), GAIA +3.80 (43.42 vs 39.62 for SAO), HLE +0.12 (10.03 vs 9.91 for SAO).
- Automation Bench pass@1: 13.33 vs 12.06 for SAO at 4B, and 15.00 vs 14.20 for SAO at 30B-A3B. The untrained base models scored 0.00 and 0.17.
- Claw-Eval pass^3 (success on all three trials): up 0.95 points at 4B and 1.12 points at 30B-A3B, per the paper's text.
The HLE gains are small, especially at 30B-A3B. The search gains on BrowseComp-Plus and GAIA are the clearest part of the table.
More attempts. In Figure 6, the 4B MPO model leads GRPO and PPO at every tested number of attempts on all three search benchmarks. At 32 attempts it reaches 75.90 on BrowseComp-Plus, 69.90 on GAIA and 34.00 on HLE, which is 3.61, 5.82 and 5.80 points above the better of GRPO and PPO. The paper says the 30B-A3B gains with more attempts depend on the benchmark and the budget.
What the ablations say
- Turn ratios matter. Swapping the turn-level ratio for token-level ratios ("MPO w/ token IS") lowered every reported score. At 4B, search pass@1 fell by 2.36, 3.12 and 1.03 points on BrowseComp-Plus, GAIA and HLE.
- Turn credit alone helps. With beta at 0 (no token adjustment), the turn-only variant still beat standard PPO on all six search metrics at 4B, by 0.95, 2.49 and 0.66 pass@1 points (Table 3 vs Table 1).
- Token adjustment helps more. Compared with that turn-only variant, full MPO (beta 1.0) added 6.28, 5.27 and 2.04 pass@1 points. Pushing beta to 1.5 or 2.0 lowered scores again.
- Decay settings need tuning. A turn decay of 0.80 beat 0.60 and 0.85 in Table 2, and a token decay of 0.95 beat 0.90 and 1.00 on all six metrics.
Caveats
- One run per setting. The paper says each method has one training run per model size, and each ablation setting has one run. Scores come from the best checkpoint of each run, averaged over five evaluations. That reduces evaluation noise, not training-run noise.
- Private training data. The Claw-Eval and Automation Bench training sets are "proprietary internal company datasets," per Appendix B. The search setup follows a published recipe (SUPO).
- No code link. The arXiv v1 paper does not link code or weights, as of October 29, 2026.
- Modest scale. The main comparisons trained for 200 to 300 steps per setting on 4B and 30B-A3B models. The 4B training curves in Figure 4 run to step 600, but the paper does not test larger models.
Why it matters
Group-based recipes such as GRPO give a whole rollout one score. This paper adds to the case that agents need a critic and a credit unit that matches how they act: one turn at a time. The change sits in the training math: a different way to compute advantages and importance ratios, with linear-time cost per trajectory, per Appendix B. If later work with released code reproduces the search gains, turn-level credit could become a default for training long-horizon agents.
