Surge AI's new DAYJOB: Tax benchmark asks a blunt question: can an AI agent do the job of a tax professional, not just answer a tax question? Its answer is mostly no. The best model on the leaderboard, Claude Sonnet 5.5, passes 10.8% of attempts. Over the past two days the result moved from Surge's own accounts on X to a long Hacker News thread, where the reaction split into three camps: people who say AI already catches their accountant's mistakes, people who say the test is unfair to models without a tax-specific harness, and tax professionals who say the real limit is liability and judgment.
Surge's blog post is dated October 27, 2026. Surge founder @echen announced it on X at 2:11 PM PT that day, and the Hacker News thread went up at 8:46 AM PT on October 28. Counts below were checked at about 7:20 AM PT on October 29: the @echen post had 59 likes and about 8,170 views, and the Hacker News thread had 247 points and 157 comments.
What the benchmark tests
DAYJOB: Tax is the third release in Surge's DAYJOB family, after Healthcare and Finance. According to Surge, it has 50 assignments ("environments") across eight areas of tax practice: individual and family tax, small-business tax, partnerships, property and investments, estates, gifts and trusts, international tax, corporate transactions, and tax review and disputes. The assignments come with 1,789 workspace files, a median of 30 per task and as many as 123. Most are PDFs (1,487), plus 117 Excel workbooks, 38 Word documents, and emails, images, CSVs and other records.
The agent gets a client request and the records, then has to produce a return, workpapers, a review or written advice. Surge reports pass@1 at a 95% rubric threshold: an attempt passes only if it meets at least 95% of the task's grading criteria, which cover calculations, supporting evidence, "consequential issues" and the finished deliverable. Surge says tax experts built and reviewed each task, with three layers of review.
The point, Surge says, is that most tax benchmarks hand the model a defined problem. DAYJOB asks the agent to work out what the problem is, including issues the client did not raise. One task includes a photo of a handwritten receipt for resale stock. Another gives a silo-services company that thinks it only operates in Alabama, while its dispatch log shows crews in five states.
The scores
On Surge's leaderboard, as of October 29, 2026:
- Claude Sonnet 5.5 (Adaptive/Max): 10.8%
- Claude Fable 5.1 (Adaptive/Max): 9.6%
- Claude Opus 5.5 (Adaptive/Max): 9.2%
- Muse Wonder 1.0 (Max reasoning): 6%
- Grok 4.7 (xHigh reasoning): 5%
- GPT-6 Astra (Max reasoning): 4.8%
- GPT-6.1 Sol (Max reasoning): 4%
- Muse Spark 1.3 (Max reasoning): 3.6%
- Gemini 4 Argon (High reasoning): 3.2%
- Hy4 Preview (High reasoning): 1.6%; DeepSeek V4.1 Flash 1017 (Max reasoning): 1.2%; Gemini 3.8 Flash (High reasoning): 0.8%; GLM 5.3 (Max reasoning): 0.4%
- Kimi K3, Mistral Large 4 and Qwen 3.8 Max: 0%
The example everyone repeated
Several X posts, including Surge's own, retold the same case. A client named Anna moved out of her condo in August 2024 and asked whether closing the sale in October 2026 or January 2027 was better for taxes. Both dates qualify for the home-sale exclusion of up to $250,000 in gains. What a good adviser should also say, per Surge, is that she could lose eligibility if the sale slips past about the end of August 2027. Surge says Opus caught that deadline in all five attempts and Astra in none.
A second case cuts the other way. A separating couple asked how to split a Roth IRA and a 401(k). Opus 5.5 assumed in all five attempts that they were not legally married, even though their records showed they had previously lived together in Texas, which recognizes informal marriages under some conditions, and a ceremony in Colorado. Astra said to confirm marital status first in four of five attempts.
@echen leaned on the first case in the launch post and joked that it might justify an Opus bill. The post opens with "too many benchmarks are too sanitized" and adds "but figuring out the problem is half the job and all the fun." The company account @HelloSurgeAI led with the five-state silo company and asked "Would an AI agent notice?"
@AiDevCraft put the two cases side by side: "Opposite blind spots. Run two agents and diff." Replying to @echen, @RaghavMehtaM1 wrote that "grading whether it warns you about stuff you didn't ask is way harder than grading the answer," and @aryakanna9 said agents "answer the question asked and miss the thing that actually costs money." Logan Kilpatrick (@OfficialLoganK), who works on Gemini, replied that models are getting good at sanitized tasks but the real world is messy.
Praise from people who already do this
Several Hacker News commenters described doing their own tax work with AI, and some said the models outdid or caught errors by their paid preparers.
waiwai933, a US citizen living abroad, wrote that Claude found real errors in their self-prepared return, including a wrong exchange rate, but also "provided highly inaccurate estimated tax calculations." Their takeaway: "it is a very useful tool for tax, but perhaps unsurprisingly, it is not good enough to dump all your paperwork into and expect a shiny return you don't need to check."
arpinum said "GPT Sol did better than my specialist accountant this year," with "a lot of steering from me and about 2B tokens." In a follow-up they said it filled out 59 pages of returns and supplement forms and also filed their FBAR. lexandstuff described preparing a return with Astra and Codex, then having Codex check the accountant's version: "it was riddled with errors," with a refund "nearly $2k lower than it should have been." idopmstuff replied that Claude flagged their former CPA amortizing acquired goodwill over 5 years instead of 15.
aprilthird2021 answered that the anecdotes "clash heavily with the results of the benchmark," and mlyle replied that the benchmark seems built around edge cases and whether the agent raises what the client did not ask.
The complaints
The main technical objection was the harness. licyeus wrote that they could not quickly find what tools or system prompts the agents got, and that the Astra miss "seems easily fixed by scaffolding ensuring that requirements are compiled first, then checked." bluejay2387 argued that dumping documents on a general-purpose agent "doesn't really say anything," and that the interesting question is what models do "in a well engineered harness focused on taxes." asadotzler replied: "How do you think 99.95% of people who do taxes with gen AI are doing it?"
Others questioned the numbers. pawelduda was surprised Sonnet 5.5 beat Opus and Fable, and WhitneyLand replied that "the difference is well within the error bars." andriy_koval asked for the human baseline; Surge's post does not give one. aprilthird2021 noted that publishing the Anna example risks future models learning that exact task. ilmalan was shorter, calling it "the 'dude trust us' bench" in a separate comment. On X, @mpsmyth71 flagged that it is "One vendor's test; Surge has worked with Anthropic, whose models lead."
Privacy came up too. matkoniecz wrote that "Sharing all my financial info with LLM companies is not something that I like."
What a CPA in the thread said
rbhelt, who says they are a CPA with 20+ years of experience, gave the most specific professional view. They wrote that LLMs "can be very good at preparing taxes but I would not trust them to complete a tax return start to finish," and that in a typical firm an LLM could replace "the unlicensed preparers/interns/early career stage CPAs" who fill in returns for senior review. fn-mote pushed back, pointing to posters whose accountants made large errors that LLMs caught: "Maybe CPAs with LLM assistance."
Liability ran through the thread. raegis wrote that the IRS holds the filer responsible for everything on the return regardless of the preparer, and skinfaxi added that in the US the individual must sign their own return. gamblor956, who says they began their career fixing other advisers' mistakes, predicted the next generation of tax lawyers "will get to do the same except this time the former advisers will be llms."
What it adds up to
The reactions do not really contradict the benchmark. Users who report success describe close supervision, many documents and their own checking, which is a different job from the unattended, end-to-end work DAYJOB grades at a 95% bar. The fair criticisms are concrete: no human baseline, small gaps between the top three, a vendor-run test, and no tax-specific harness. Surge says the full benchmark is available on request and has published 2 sample tasks in a dataset on Hugging Face. For now, the practical reading from both sides of the thread is the same: use an agent as a second reviewer, and keep a human who signs the return.
