ScarfBench asks what IBM calls a practical question: "Can an agent take an unfamiliar application, migrate it across framework boundaries, and leave behind software that still works?" On October 27, IBM Research and Kaggle launched a ScarfBench leaderboard with results for 10 model and agent harness pairs. IBM's main finding is that behavior, not compilation, separates the agents.
What ScarfBench measures
ScarfBench (Self-Contained Application Refactoring Benchmark) is an open IBM benchmark that IBM first introduced earlier in 2026. It covers migrations across three Java frameworks: Jakarta EE, Quarkus and Spring. For each task, the migrated app has to build, start and keep behaving as expected, tested on both focused workloads and whole applications.
IBM's post explains why this is not a single-file job. A framework move can touch dependency injection, persistence, database access, web endpoints, build systems, framework descriptors and deployment setup. An agent can produce a migration that "looks plausible, or even compiles, while the application still fails when deployed or exercised against its expected behavior," in IBM's words.
One detail matters for reading the results. ScarfBench scores an agentic system, not a bare model. Each entry is a model paired with a harness such as Claude Code, Codex or Antigravity, and IBM says the results "should therefore be read as model/harness results, not as measurements of the underlying model alone."
What Kaggle adds
According to the IBM Research post, Kaggle consulted on the evaluation design, independently reproduced and ran the benchmark on its own platform, sponsored the compute, and will maintain the benchmark as new models are released. The public results live on the ScarfBench page on Kaggle Benchmarks. Public model submission "is not available at launch but is on the roadmap," IBM says. The work is part of a wider IBM and Kaggle enterprise AI collaboration.
The ranking
Across 204 migration tasks, these are the smoke-test scores IBM reported. The smoke score is the leaderboard metric. It gives partial credit and weights larger test suites more.
- Gemini 4 Argon with Antigravity: 75.2%
- Claude Opus 5.5 with Claude Code: 70.5%
- Claude Opus 5 with Claude Code: 69.8%
- GPT-6.1 Sol with Codex: 68.4%
- Gemini 3.8 Flash with mini-swe: 66.0%
- GPT-6 Astra with Codex: 65.4%
- Qwen 3.8 Max with qwen-code: 63.7%
- Kimi K3 with kimi-code: 61.1%
- Gemini 3.1 Pro with OpenHands: 39.5%
- Gemma 4 26B-A4B with mini-swe: 24.5%
The order changes on a stricter measure. Claude Opus 5.5 with Claude Code had the highest all-tests-pass rate at 72.1%, narrowly ahead of Gemini 4 Argon with Antigravity at 71.1%. IBM's reading: the Argon pair earned the most partial credit, while the Opus 5.5 pair turned slightly more tasks into fully green test suites. Google's model placed sixth on a different coding test; see our look at Gemini 4 Argon on Google's Android Bench.
IBM ran a sensitivity check. Four tasks got zero behavioral-test credit from every pair. Dropping them raised scores but kept the top three in the same order: 79.4%, 74.5% and 73.7%. Even so, no configuration fully passed more than about three quarters of the migrations.
Builds are not the hard part
IBM calls the gap between code that builds and software that behaves correctly "the clearest result." After the four all-zero tasks were removed, three pairs compiled 100% of tasks, and 8 of 10 compiled 97% or more. Smoke-test scores, by contrast, spanned more than 50 points across the field.
IBM classified 745 failing runs:
- 535, about 72%, were runtime behavioral failures: the app built, often booted, then failed validation. These included 304 content or assertion mismatches, 210 lost routes that returned 404, and 21 explicit HTTP 500 or 405 errors.
- 152, about 20%, were compile or packaging failures.
- 58, about 8%, were boot or dependency-injection crashes.
IBM gives examples. In one Quarkus to Jakarta task, Claude Opus 5 with Claude Code scored 0 of 6 after an endpoint returned "File not found" and a test expecting HTTP 405 failed. In a Jakarta to Spring task, Gemini 3.8 Flash with mini-swe left the source app's JSF/JoinFaces wiring in place, and the Spring Boot app failed at startup. In a Spring to Jakarta task, GPT-6 Astra with Codex left a packaging type the target Liberty runtime did not support.
Turns and cost
Agents also worked very differently. On the 200-task set, median turns ranged from 32 for Claude Opus 5 with Claude Code to 79 for Gemini 3.8 Flash with mini-swe. Claude Opus 5.5 with Claude Code used 64.
IBM also published the cost each harness reported for its own runs. Only Claude Code, Codex, mini-swe and kimi-code emitted per-run cost; Antigravity, qwen-code and OpenHands did not. IBM does not say how those figures are billed or which price list they use, so read them as spend reported by each harness, not as a subscription price.
- GPT-6.1 Sol with Codex: $1.25 per solved task ($0.64 per successful run)
- Gemini 3.8 Flash with mini-swe: $1.59 per solved task ($0.73 per successful run)
- Kimi K3 with kimi-code: $3.21 per solved task ($1.27 per successful run)
- Claude Opus 5 with Claude Code: $3.83 per solved task ($1.76 per successful run)
- GPT-6 Astra with Codex: $6.65 per solved task ($3.55 per successful run)
- Claude Opus 5.5 with Claude Code: $10.10 per solved task ($5.98 per successful run)
That puts GPT-6.1 Sol with Codex fourth on score but cheapest per solved task among pairs that reported cost. For more on where OpenAI's models run for coding, see our guide to GPT-6.1 Terra in Copilot and Codex and when to pick Sol.
Why it matters
IBM built ScarfBench on the premise that coding agents in enterprise modernization should be judged on "working software." ScarfBench gives buyers a test that checks running behavior, not just a green build, and Kaggle's role means the runs were reproduced outside IBM. IBM says Kaggle will add new models as they are released.
Two cautions apply. The scores rate a model and a harness together, so changing either one may change the result. And the open-weight Gemma 4 26B-A4B entry placed last at 24.5%. Readers comparing local options can start with our list of the best open-source LLMs for coding.
