On Nov 8, 2026, Y Combinator CEO Garry Tan posted early results from an open source test of AI coding harnesses, the software that wraps a model like Claude Opus 5.5 or OpenAI's GPT-6.1 Sol and turns it into an agent. @garrytan wrote at 10:09 AM PT: "Out of curiosity I am doing some open source eval work on harnesses with frontier models and here's what is coming back so far" with a screenshot of results. As of 5:40 AM PT on Nov 9, the post had 513 likes, 145 replies and about 104,000 views.
Seven setups passed 71 to 79% of the same 20 coding tasks, so the test could not separate them. Cost per solved task ranged from $0.17 to $0.44. Replies argue about the test itself, about missing harnesses and about cost.
What is an AI coding harness?
A harness is the software wrapped around a model: its system prompt, its tools and its memory. The same model can behave differently under different harnesses.
Tan's repo, garrytan/g-harness-evals, holds the model (Opus 5.5 or GPT-6.1 Sol) and the tasks fixed, swaps only the harness, and measures pass rate, cost per solved task and time. In a follow-up post he said he is "doing this in public".
The README calls the 20 tasks "public multi-user tasks". It describes the output as "dev results on public tasks at the pinned versions, not held-out confirmation" and says the cost differences are "descriptive, with no test". Tan's screenshot calls them "coding tasks", but the README says the task world comes from gbrain-evals, another Garry Tan project, and the chart Nous Research posted (below) describes them as office tasks in a mock company Slack.
What did the harness eval find?
The screenshot states: "We gave 7 setups the same 20 coding tasks." All of them passed 71 to 79% of the time. The setups were Claude Code, OpenClaw, Hermes and QM on Opus 5.5, plus Codex, OpenClaw and Hermes on GPT-6.1 Sol.
Tan wrote: "Every comparison came out 'can't tell apart', because 14 of the 20 tasks scored the same in every setup. So the test was too easy to separate them, and that's a test problem, not a tie." He added that the harder rerun on Terminal-Bench "is running now".
Pass rate point estimates, per the README (which also gives 95% ranges): on Opus, QM 72.5%, Claude Code 76.3%, Hermes Agent 78.8%, OpenClaw 76.3%. On Sol, Codex 71.3%, OpenClaw 73.7%, Hermes Agent 75.0%. The README says no tested pair could be told apart.
Disclosures in the README: the repo and evaluation are run by Garry Tan; QM is a project of Y Combinator; Y Combinator is an investor in Hermes; OpenAI is now the key benefactor of OpenClaw, and OpenClaw's creator, Peter Steinberger, works for OpenAI.
Which harness is cheapest?
On Opus 5.5, cost per solved task: OpenClaw about $0.44, Hermes $0.24, Claude Code $0.22, QM $0.19. On GPT-6.1 Sol, Tan's screenshot calls Codex "the cheapest and fastest at $0.17". In the README's ranking, though, the cost ranges overlap, so every harness shares rank 1 on its model; OpenClaw on Opus clears only Claude Code's and QM's ranges. The README gives OpenClaw on Sol a headline cost of $0.22 and Hermes on Sol $0.19.
Median time per trial: Codex 30.1 s, Claude Code 40.1 s, QM 40.1 s, Hermes on Opus 64.6 s, OpenClaw on Opus 99.4 s.
Tan wrote that "A second, separate run found the same pattern." His conclusion: "For now, choose on cost and speed: Claude Code on Opus, Codex on Sol."
About 81% of OpenClaw's extra cost over Claude Code on Opus is cache writes, per the README, because it re-sends a much bigger system prompt and bigger tool results.
Is Claude Code or OpenClaw better?
On these 20 tasks, the test cannot tell them apart. Both scored 76.3% on Opus. The difference is cost and time: Claude Code $0.22 per solved task at a 40.1 s median per trial, OpenClaw about $0.44 at 99.4 s.
@NousResearch, which makes Hermes, replied "Kinda looks like Hermes is smarter Garry" with a chart showing Hermes on Opus 5.5 at 78.8%, the highest point estimate in the set. The chart's caption describes the test as "20 office tasks in a mock company Slack, 4 runs each (560 runs)." The README, though, says pass rates could not be told apart in any tested pair.
Tibo (@thsottiaux), whose X bio lists Codex and ChatGPT at OpenAI, wrote: "Thanks for running those and publishing" (239 likes as of 5:40 AM PT on Nov 9).
Why do people say the test was too easy?
@ArturSkowronski: "The number I'd look at first is 14 of 20 tasks scoring the same everywhere. That says more about the task set than about the harnesses."
@ZKiihne, who says his company builds API simulators and is in the current YC batch: "I would naively assume that there is not much differentiation because the tasks are not hard enough."
@the_panwright: "The hard part is making real tasks to evaluate. A task that costs $0.44 is probably quite small." The reply adds that people care about larger tasks and asks for tests of work that would take hours or days.
@0xDevShah quote-posted: "these are only marginally better than the other. we've hit the ceiling of current harness engineering, and there is a great need to design fundamentally different architectures." (24 likes as of 5:40 AM PT on Nov 9)
Which harnesses are missing from the test?
@someRandomDev5: "The top 3 mainstream omissions I can think of off the top of my head:" Cursor, Pi and DeepSeek Harness. He added: "Really weird to pretend that OpenClaw is a coding harness while skipping out on these top-tier harnesses." Tan answered: "I'm on it". The README lists QM as a "Pi loop" harness, but Pi itself was not a tested setup.
Others named tools not tested: @revezsilva asked "What about OpenCode?" and @TheCoderBtw asked "Where is droid?" Droid is Factory's coding agent; see our report on Factory Droid 1.0 and its new agent harness.
Is cost the real story?
@shal0vbxm: "can't tell which harness is smarter yet. can tell openclaw is burning 2x the tokens and 2x the time on the same opus." The README does give OpenClaw about twice the cost of the other Opus harnesses and 99.4 s per trial against 40.1 s for Claude Code. It puts about 81% of the extra cost over Claude Code on cache writes and gives no total token count for Opus.
Do model-specific harnesses beat multi-model ones?
@rahulrajaramio: "This aligns with my experiments as well. The one conclusion I would draw is that multi-model harnesses are generally less capable than model specific harnesses. The latter optimize the context window for the models to which they cater. Claw-like harnesses are less effective."
@SvenMeyer replied to him: "Wrong. In most harness benchmarks I have seen Pi comes out first, even better with Opus / Sonnet than Claude Code itself."
What should the next harness eval measure?
@Alicechen8888: "One metric I'd add is how many times a human has to step in before the patch is ready to merge. Two setups can pass the same tests and still take very different amounts of babysitting."
@KrrishKumar2028 quote-posted: "The number I'd add is cost per false "done". A cheap harness that confidently ships a broken fix gets expensive in review."
@MarcusSpillane: "Most harnesses look fine for a few tool calls, then drift off their own constraints on long runs. Are you scoring 50+ step tasks separately?"
What to watch next
- The Terminal-Bench rerun, which Tan said on Nov 8 was running.
- Whether Cursor, Pi, DeepSeek Harness, OpenCode and droid get added, after Tan's "I'm on it".
- Your own numbers: run a harness on your own tasks and track cost per solved task. Public task sets saturate, and the README says these results are not held-out confirmation.
No Hacker News thread on the eval was found as of 5:40 AM PT on Nov 9.
Sources
- Garry Tan's original post with results screenshot: https://x.com/garrytan/status/2119492448200405110
- Garry Tan's follow-up linking the repo: https://x.com/garrytan/status/2119513847048618311
- Evaluation repo and README: https://github.com/garrytan/g-harness-evals
- Garry Tan's reply on missing harnesses: https://x.com/garrytan/status/2119623568686522623
- Tibo (@thsottiaux)
- @NousResearch
- Nous Research Hermes Agent repo
- @someRandomDev5
- @revezsilva
- @TheCoderBtw
- @ArturSkowronski
- @ZKiihne and follow-up
- @the_panwright
- @0xDevShah
- @rahulrajaramio
- @SvenMeyer
- @Alicechen8888
- @KrrishKumar2028
- @MarcusSpillane
- @shal0vbxm
