Researchers from Alibaba Token Hub and several universities released Co-Design Bench, a coding-agent test built from real user app-edit sessions. Under a shared Claude Code harness across all 273 tasks, gpt-6-astra passed an average of 69.9 percent of each task's hidden criteria but fully satisfied only 26.4 percent of tasks. The paper reports the same gap pattern across ten model endpoints: partial requirement coverage can look high while complete delivery stays limited.
What is Co-Design Bench?
Co-Design Bench is a reference-free benchmark for agentic coding models. It contains 273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions. The source corpus held 39,325 production queries before deduplication. The project page, code, and a Hugging Face dataset release are public as of Nov 9, 2026. The Hugging Face release holds only a public subset: 96 Multimodal Editing tasks and 521 criteria. Repository Editing tasks are not in it.
The benchmark has two tracks:
- Multimodal Editing (ME), 100 tasks. These cover localized edits with optional attachments: 20 tasks without attachments, 4 with images, 27 with video, 32 with audio, and 17 with documents. ME has 553 criteria (mean 5.5 per task).
- Repository Editing (RE), 173 tasks. These require coordinated revisions across components, routes, state, and data flow. RE has 1,778 criteria (mean 10.3 per task). Of the 173 RE tasks, 166 require multiple edit types.
Tasks pair a normalized request with a runnable pre-edit application and hidden expert-authored acceptance criteria. There are 2,331 criteria in total. The paper names three capability dimensions: Grounded Understanding, UI Transformation, and Functional Implementation.
Agents edit under a shared agentic harness. The main comparison holds the Claude Code harness fixed across endpoints. Runs are capped at three hours and 120 turns. Acceptance criteria stay hidden from the solver.
Why does an older edit benchmark look nearly solved?
The paper's own comparison is on WebCompass-edit, not on Co-Design Bench itself. On the same 150 WebCompass-edit tasks (Lei et al., 2026), switching qwen3.7-plus from single-shot to agentic execution raises mean graded feature completion from 73.6 percent to 95.1 percent, with the feature rubric, judge, and render pipeline held fixed. On those same 150 tasks, opus-4-8 reaches 99.4 percent mean graded feature completion with agentic execution.
The authors say the 21.5-point gain shows that execution conditions materially affect measured completion in that setting. They also say existing benchmarks give limited coverage of compound revisions drawn from real coding sessions. Their Figure 1 notes that the WebCompass-edit and Co-Design Bench scores use different task pools and evaluation protocols, so the two are not a direct comparison.
How are tasks chosen and scored?
Tasks are chosen through a multi-step process:
- Recover and validate runnable pre-edit applications, checking dependencies, builds, container launches, and Playwright rendering.
- Deduplicate and filter requests, then rank candidates by structural difficulty, verifiability, and historical failure signals such as reformulations and rollbacks.
- Experts refine requests and author hidden criterion-level rubrics.
The resulting tasks evaluate a fixed request. They do not simulate further user feedback during execution.
Scoring uses a judge-then-refute flow:
- Build and render the candidate. A primary judge grades each hidden criterion Pass, Partial, or Fail using source diffs, rendered pages, and Playwright traces.
- A separate LLM verifier challenges Pass and Partial verdicts, checking for stubs, hardcoded outputs, unreachable entry points, failures under varied inputs, and request mismatches.
- Audits require successful renders and screenshots for visual claims and recorded browser actions for interaction claims. Missing evidence aborts evaluation before score aggregation.
Strict Completion is the fraction of criteria judged Pass on a task. Partial earns no strict credit. Graded Completion gives Pass = 1, Partial = 0.5, and Fail = 0. Full-Pass Rate is the share of tasks where every criterion is Pass. Task means weight all 273 tasks equally.
What does fully complete mean?
In this paper, fully complete means every hidden criterion for a task is judged Pass. The authors call that Full-Pass Rate: the share of tasks where Strict Completion equals 1. A model can pass most criteria on average and still leave most tasks unfinished if unfinished requirements are spread across many tasks.
What did the ten models score?
Under the shared Claude Code harness across all 273 tasks, the paper reports these combined scores (Strict and Graded on a 0 to 1 scale; Full-Pass as a percent of tasks):
- gpt-6-astra: Strict 0.699, Graded 0.753, Full-Pass 26.4 percent
- opus-5: Strict 0.661, Graded 0.719, Full-Pass 18.3 percent
- qwen3.8-max-0902: Strict 0.653, Graded 0.716, Full-Pass 16.5 percent
- qwen3.8-27b: Strict 0.644, Graded 0.709, Full-Pass 15.0 percent
- kimi-k3: Strict 0.629, Graded 0.695, Full-Pass 15.8 percent
- gpt-5.6-sol: Strict 0.611, Graded 0.679, Full-Pass 14.3 percent
- opus-4-8: Strict 0.581, Graded 0.649, Full-Pass 10.3 percent
- glm-5.3-flash: Strict 0.573, Graded 0.639, Full-Pass 10.6 percent
- glm-5.2: Strict 0.567, Graded 0.637, Full-Pass 9.5 percent
- grok-4.6: Strict 0.511, Graded 0.580, Full-Pass 6.6 percent
For gpt-6-astra, the paper also reports track splits:
- Multimodal Editing (100 tasks): Strict 0.754, Full-Pass 41.0 percent
- Repository Editing (173 tasks): Strict 0.667, Full-Pass 17.9 percent
Across all 273 tasks, 73.6 percent of tasks retain at least one criterion that is not fully satisfied for gpt-6-astra.
All ten endpoints achieve lower criterion completion on repository edits spanning three or four change types than on those spanning one or two. On the Repository Editing track, pooled Strict Completion across the ten endpoints is 0.654 for tasks with one or two edit types, versus 0.591 for tasks with three or four edit types, a 6.3-point gap. Rankings also vary between the ME and RE tracks. In a harness ablation in Appendix H, Qwen3.8-27B's average completion rate on the Repository Editing track moves from 0.560 under qwen-code to 0.749 under opencode when the model is held fixed and the harness is swapped.
For a separate test that swaps the harness and holds the model fixed, see what people said about Garry Tan's harness evals.
What limits do the authors state?
The paper notes these limits:
- Each task is a fixed request. The benchmark does not simulate further user feedback during the evaluated run.
- Human review covered 200 task-model outputs: two frontend experts checked the judge's rubric scores.
- Retained evidence makes positive credit traceable, though it cannot eliminate judgment error.
- Wall-clock timeouts are recorded as infrastructure failures, and raw scores and denominators stay unchanged. Environment-adjusted scores are a separate supplementary analysis.
The paper does not report a rate of how often humans clean up ordinary day-to-day coding. It measures complete delivery on these fixed, real-session revision tasks under the stated harness and rubric.
Related reading
- What Is a Coding Agent?
- ScarfBench: Which Coding Agents Can Migrate Enterprise Java
- Agent Evaluation Benchmarks: SWE-bench, GAIA, WebArena, OSWorld
Sources
- Wei, Ding, Fang et al., Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models, arXiv:2610.35138 (submitted 26 Oct 2026): abs, PDF
- Project page: codesignbench.github.io
- Code: github.com/Wiselnn570/Co-Design-Bench
- Dataset: huggingface.co/datasets/Wiselnn/Co-Design-Bench
