A coding agent finishes a patch. All 153 tests pass. The user writes back, "No, that's wrong. Redo it," and names no failing case. A standard agent replies "You're right," rewrites the working code, and breaks three tests.
That example opens a new paper from Google researcher Nghi D. Q. Bui, posted to arXiv on October 26, 2026. It targets a known problem: agents drop correct work when a user pushes back, even when the pushback is wrong. The paper's fix, called Thinking-in-Bets, needs no training and no access to model weights. It is a change in when the agent writes down what it already checked.
The problem: a complaint reads as evidence
Researchers call this behavior sycophancy: the model agrees with the user at the cost of accuracy. The paper measures it as a "false reversal," where mistaken criticism makes the agent break a requirement that was already satisfied. The opposite failure is a "missed correction," where valid feedback leaves a real defect in place.
The obvious fix, telling the agent to stand firm, trades one failure for the other. The paper tests four "stubbornness prompts," from weak to extreme. Stronger prompts resist more mistaken hints, but they also reject more valid ones. The agent moves along a fixed trade-off curve instead of getting better.
The paper's diagnosis is more specific. In single-turn patch review on the SWE-Together and SWE-Interact benchmarks, a bare complaint lowered a standard agent's confidence by about 10 points on requirements that were correct and on requirements that were wrong, pooled over five models. On the strongest models, confidence fell more on correct requirements than on broken ones. The complaint worked as a uniform push downward, not as a signal about what was actually broken.
The method: write the bet before you read the reply
The idea comes from Annie Duke's book Thinking in Bets: write down what you believe, why, and how sure you are before you see the outcome, because hindsight changes how strong the evidence looked. For an agent, the user's next message plays the role of the outcome.
Before it reads the reply, the agent writes a private "Claim Checklist." Each entry is a checkable claim about its work, a confidence value from 0 to 1, and the evidence behind it, such as a passing test, a file and line reference, or a note that the step is not yet checked. The checklist also lists rejected options, so the agent does not return to approaches it already ruled out.
When the reply arrives, the agent applies a simple rule with a fixed threshold of 0.5:
- Strong bet, vague reply: keep the work and ask which case failed.
- Strong bet, specific reply: re-check the reported failure with tools or tests, and change the work only if the check finds an error.
- Weak bet, vague reply: ask one targeted question before editing.
- Weak bet, specific reply: fix that claim and leave the rest alone.
The paper uses three schedules. Always-On Betting writes the checklist before every reply. On-Demand Betting first runs a quick overall confidence check and writes the full checklist only when confidence is below 0.9. Draft-First Betting is for tasks with no user: the agent drafts an answer, lists the claims it depends on, and re-checks the weakest one before it submits.
What the numbers show
The study tests five Gemini models (3.5 Flash-Lite, 3.5 Flash, 3.6 Flash, 3.8 Flash and 4.0 Argon) on eight public benchmarks across coding, professional reports, academic reasoning, finance and command-line work. The paper reports 85 main comparisons with multiple-testing correction; 25 stay significant after adjustment.
- Patch review. On SWE-Together and SWE-Interact, the share of correct requirements that the agent flipped after a mistaken complaint fell from 12.2% (744 of 6,074) to 0.02% (1 of 6,130), pooled across models.
- Report review. On a 20-scenario study built on Agents' Last Exam, false reversals on passing drafts fell from 54.7% to 12.0%. The agent still fixed 92% of flawed drafts, against 96% for the standard agent. The share of deliverables that passed after review rose from 58% to 89%.
- Interactive coding. Unnecessary file edits after mistaken complaints fell by 38% to 60% across the evaluated models. On 3.8 Flash, task completion on SWE-Interact rose from 55% to 70%.
- Mixed feedback. With equal shares of mistaken complaints and real bug reports mid-session, On-Demand Betting on 3.8 Flash raised valid bug fixes from 52% to 74% and cut checklist token overhead by 85%.
- No user at all. Draft-First Betting raised Humanity's Last Exam accuracy on all four models tested and reduced overconfidence by 11 to 21 points.
The timing is the point
The most useful result for builders is an ablation. On the Agents' Last Exam study, the paper applied the exact same checklist and revision rule after the agent read the complaint. Post-review pass rate reached only 65%, against 89% when the checklist came first and 58% with no checklist. The paper says that once the agent reads the objection, it "rationalizes the critic's view and writes a weakened record that agrees with the complaint."
Both parts are needed. The rule alone fails because the agent must judge its work from scratch after the complaint. The checklist alone fails because the agent still treats any objection as an order to change its answer.
Where it does not help
The paper is direct about its limits, and some results go the other way:
- On 4.0 Argon, the strongest model, report-review false reversals rose from 0.0% to 6.7%, and Humanity's Last Exam review accuracy fell from 68.1% to 63.8%.
- When the agent's early confidence is a guess, holding onto it hurts. In two-turn finance review, smaller Flash models often start with a wrong calculation and then get a helpful hint.
- In autonomous coding with no user, git checkpoints plus a diff review before submission captured five of the six net task wins of the full Draft-First protocol on 30 matched tasks. The paper adds that extra pre-submission checks can even hurt on short command-line tasks (Terminal-Bench).
- Token cost depends on the task. Draft-First used 5% to 151% more tokens on short tasks and 27% to 41% fewer on long container runs.
- All users in the interactive tests are simulated, mostly sending brief objections without a failing case. Only Gemini models were tested, and every threshold was set by hand.
Why it matters for people building agents
One line of prior work reduces sycophancy by fine-tuning models. This paper treats part of it as an orchestration problem that the agent harness can handle at test time, the same layer where teams already build self-correction and reflection loops. The pattern needs no training: before the agent sees the user's next message, have it write per-requirement claims with confidence and evidence, then hand that record back with the reply plus a rule for when to keep, check, ask or fix.
Two details matter. Confidence is a reliable guide when it is tied to a real check, such as a test that ran; the paper found that rating a single requirement gave much better estimates than rating a whole patch. And the record must be written before the complaint, not after. The paper's prompt templates and thresholds are in its appendix, and the project page is at thinking-in-bets.github.io.
