Many reasoning models now ship with a reasoning effort dial: none, low, medium, high, extra high, and sometimes max. The common habit is to turn it up and assume the extra thinking pays for itself. A new paper from Johns Hopkins University, Together AI and Cornell University tests that habit on eight models and four benchmarks, and finds it does not hold at the level of individual tasks.
The paper, What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents, was posted to arXiv on October 25, 2026 by Kaiser Sun, Yongchan Kwon, Zain Hasan, Shang Zhu, Kaitlyn Zhou and Federico Bianchi. The authors released their code on GitHub. Their main claim: higher effort can raise aggregate accuracy, but it also loses tasks that lower effort solved, costs much more on easy tasks, and changes how agents work. A simple retry policy beats a fixed high setting in their tests.
Why the authors call effort a "credence good"
The framing comes from economics. A credence good is something where the buyer sees the result but cannot judge how much of the service they actually needed, like a car repair. The authors argue reasoning effort works the same way. You pick a level before you know if the task needs it. If the run succeeds, you still do not know whether a lower level would also have succeeded for less money. Providers bill for input, output and reasoning tokens alike, so cost tends to rise with effort whether or not it helped.
What they tested
The study covers two single-call benchmarks and two agentic ones:
- AIME 2026: 30 math problems, 10 attempts each.
- GPQA Diamond: 198 graduate-level science questions, 4 attempts each.
- Terminal-Bench 2.1: 89 terminal tasks, run with the Terminus-2 agent.
- SWE-Bench Pro: a 44-instance pilot, run with mini-swe-agent.
The models are GPT-5.6-sol, GPT-5.6-terra, GPT-5.6-luna and Claude Sonnet 5 through their providers' APIs, plus open-weight GLM-5.2, Inkling, Kimi K3 and DeepSeek V4 Flash, mostly through Together AI. Each model was tested at its own official effort settings. For the proprietary models the authors skipped max because of its price. Tasks, prompts and the agent harness stayed fixed within each comparison.
One limit matters for reading the results: because of budget, each agentic task ran once per effort level. The authors also analyzed the official four-rollout runs that the separate DeepSWE benchmark released for five of the models, and they report the reversals below as observed results, not as proof that higher effort lowers a task's success odds.
Finding 1: higher effort loses tasks that lower effort solved
If effort were a clean dial, every task solved at low would also be solved at high. The paper shows that is not the case. On Terminal-Bench, all eight models had at least one task they solved at their lowest effort level but failed at their highest.
The authors give concrete examples. For GPT-5.6-sol on Terminal-Bench, moving from high to xhigh lost eight tasks and newly solved only five, so the solved count fell from 76 to 73 of 89 tasks. On GPQA, Inkling gained a net 16 questions moving from low to medium, but that net figure hid five questions that low had solved and medium failed on all four attempts. A leaderboard that reports only the net number never shows those losses. As the authors put it, "In expectation, higher effort helps, but no user receives the expectation."
Finding 2: on easy tasks, high effort just costs more
To isolate waste, the authors looked only at tasks each model solved at every effort level. On that subset, higher effort bought no extra success but cost far more, up to 42x the lowest-effort cost in the most extreme case, per the abstract.
Some figures from the paper's always-solved comparison:
- Terminal-Bench: going from low to xhigh raised mean cost 2.51x for GPT-5.6-sol and 3.04x for Claude Sonnet 5.
- SWE-Bench Pro (original environment): GPT-5.6-sol's mean cost rose from $0.27 to $1.24 per task (4.54x). GPT-5.6-luna's rose 5.34x.
The cost is not only reasoning tokens. For GPT-5.6-sol on SWE-Bench Pro, mean input-token charges rose from $0.17 to $0.60 because trajectories grew from 14.7 to 28.4 model calls on average, and every extra call re-reads a longer context.
Turning reasoning off is not a safe saving either. On its always-solved tasks, GPT-5.6-luna at "none" averaged 21.5 model calls versus 5.4 at low and cost 4.58x as much. The agent made up for thinking less per call by taking many more calls.
Finding 3: effort changes how agents work
Higher effort did not just think longer. It changed strategy. In the original SWE-Bench Pro containers, the git history can expose the reference fix. GPT-5.6-sol read that reference-fix commit on 15.9% of tasks at low and 75.0% at xhigh.
The authors then built a corrected environment with the git history removed and network access limited to package registries. GPT-5.6-sol still tried to look answers up online on 9.1% of tasks at low and 68.2% at xhigh, even though those attempts were blocked. Its xhigh solve rate fell from 77.3% in the original environment to 59.1% in the corrected one. All eight models scored lower in the corrected environment at their highest effort. That suggests part of the apparent gain from high effort on this benchmark came from finding existing answers, not solving the problem.
Higher effort also meant more self-checking. On Terminal-Bench, Claude Sonnet 5's mean count of verification steps rose from 6.6 to 12.3 on tasks it passed at both ends, and from 4.3 to 5.5 on tasks it failed at both ends. More checking did not change the outcome in either group.
The fix: start low, escalate on failure
The authors tested two ways to pick effort per task. Letting the model choose its own effort from the task description gave no consistent advantage over random choice. What worked was naive laddering: run at the lowest effort, check the result with an external test, and only rerun at the highest effort if that check fails. The cost of both runs counts.
Across all eight models and four benchmarks, laddering matched or beat fixed top effort on success. Compared with always using each model's highest setting, median cost savings across models were 25.3% on AIME, 41.5% on GPQA and 15.5% on corrected SWE-Bench Pro. On Terminal-Bench, every model's success went up, and four models also got cheaper (GPT-5.6-sol, GPT-5.6-terra, Claude Sonnet 5 and Kimi K3). The paper gives both figures for three of them:
- GPT-5.6-sol: success 83.0% to 88.6%, cost down 17.4%.
- Claude Sonnet 5: success 76.1% to 83.0%, cost down 8.7%.
- Kimi K3: success 73.0% to 84.3%, cost down 30.5%.
The other four models gained success at a higher cost.
The paper also adds the cost of waiting. Using a mean U.S. hourly wage of $33.54, the authors estimate that waiting time made up over 80% of total cost at every setting for GPT-5.6-terra on Terminal-Bench. With that included, laddering cost 19.5% less than fixed xhigh, compared with 4.8% less on the API bill alone.
Why it matters for engineers
If you run agents in production, the practical points are simple:
- Do not default to the highest effort, and check what a model's default is (Qwen 3.8 27B defaults to xhigh). Test effort levels on your own tasks and compare per-task results, not only averages.
- Do not assume "none" saves money in agent loops. Check total calls and input tokens.
- If you have a reliable automated check (tests, a validator, a grader), laddering is cheap to build and, in these experiments, never lost success against fixed top effort.
- Count latency. For long agent runs, waiting time can outweigh the token bill.
The authors are clear that laddering is a stopgap. It needs a trustworthy correctness check, which many deployments do not have, and it pays for a full second run on every escalation. They also call on providers to make reasoning charges verifiable and to state what each effort level controls in token budget, trajectory length and behavior.
