On October 26, 2026, researchers from Alibaba's Token Hub group, with co-authors from Fudan, Zhejiang University, CUHK, Shanghai Jiao Tong and NTU, posted Co-Design Bench to arXiv: a coding-agent benchmark built from real user requests instead of synthetic prompts. The Decoder picked it up on November 8, and the headline numbers look like a contradiction until you read the method.

The best of the ten models tested, OpenAI's GPT-6 Astra, passes 69.9% of a task's acceptance criteria on average, but fully satisfies only 26.4% of tasks. Same runs, two very different stories.

If you build with coding agents, or evaluate any agent that has to deliver a multi-part result, this paper is worth an hour. Not because of the leaderboard, but because of three evaluation lessons that apply to your own agent: score whole tasks, not just criteria, the harness is part of what you're measuring, and easy benchmarks saturate once the agent can run its own code.

All numbers below come from the paper unless noted. It's a preprint, not peer reviewed, and Alibaba (the lead affiliation) also makes the Qwen models in the comparison.

What Co-Design Bench is

  • Source: 39,325 production queries across 26,314 real coding sessions, deduplicated to 37,463 candidates.
  • Selection: candidates were ranked by structural difficulty, verifiability and historical failure signals: users who rephrased a request two or three times, or made four or more follow-up attempts or an explicit rollback. So the set leans toward work agents already struggled with.
  • Tasks: 273, each pairing a runnable pre-edit app, a normalized request and hidden expert-written acceptance criteria (2,331 in total).
  • Two tracks:
    • Multimodal Editing (ME), 100 tasks: localized edits, often with an attachment (27 video, 32 audio, 17 document, 4 image, 20 none). Example: pull a value from a receipt and update every dependent total.
    • Repository Editing (RE), 173 tasks: coordinated changes across components, routes, state and data flow. 166 of the 173 need more than one type of edit. Criteria per task average 10.3, with a maximum of 46.
  • Harness: every model ran inside the same Claude Code harness, same prompts and tools, capped at three hours and 120 turns.

Lesson 1: criteria pass rate and task pass rate are different metrics

The paper reports three numbers per model:

  • Strict completion: share of a task's criteria that fully pass, averaged across tasks.
  • Graded completion: same, but a Partial verdict counts as 0.5.
  • Full-pass rate: share of tasks where every criterion passes.

Here's how the top three came out across all 273 tasks:

Model (Claude Code harness) Strict completion Full-pass rate
GPT-6 Astra 0.699 26.4%
Claude Opus 5 0.661 18.3%
Qwen3.8-Max 0.653 16.5%

Grok 4.6 came last at 6.6% full pass. By track, GPT-6 Astra hit 0.754 strict / 41.0% full pass on multimodal edits, and 0.667 strict / 17.9% full pass on repository edits. In the authors' words, 73.6% of its tasks "retain at least one criterion that is not fully satisfied."

Why the gap is so wide is just arithmetic. A user doesn't receive 70% of a feature; they receive a feature that works or one they have to finish. Here's my own illustration (not from the paper, and it assumes criteria are independent, which real ones aren't): if an agent passes each criterion 90% of the time and a task has 10 criteria, the chance that all 10 pass is 0.9^10, about 35%. Per-criterion averages hide this compounding.

The paper also shows the two metrics can rank task types differently. Document-grounded edits had the highest pooled strict score (0.652) but only 18.2% full pass, versus 37.0% for text-only tasks. Functional Implementation tasks had the highest strict score of the three capability groups (0.619) and the lowest full-pass rate (10.3%).

Takeaway for your evals: if your agent eval averages rubric items, you're reporting the flattering number. Report task success (all required criteria pass) as the headline, and keep criterion coverage as the diagnostic that tells you which requirement keeps failing.

Lesson 2: the harness is part of the system under test

In an ablation, the authors held two models fixed (Kimi K3 and Qwen3.8-27B) and swapped the harness between codex 0.142.0, opencode 1.18.13 and qwen-code 0.21.11, with prompts, criteria and judges shared.

Harness Qwen3.8-27B, repository edits
opencode 0.749
codex 0.638
qwen-code 0.560

That's a 0.189 spread for the same model depending on the loop around it. Worse for leaderboard readers: the harness ordering reversed between the two models on multimodal edits (Spearman ρ = -1.00), and the Kimi-minus-Qwen gap on repository edits ranged from -0.040 to +0.122 (16.2 points) depending on the harness. Which model "wins" depended on the tool it ran in.

The authors' recommendation is to report model and harness together, and to hold the harness fixed when comparing models.

Takeaway: "Should we switch to model X?" is the wrong unit of evaluation. Test the pair you'll actually ship: model + harness + system prompt + tools. A model that wins in someone else's scaffold can lose in yours.

Lesson 3: once the agent can run its code, easy benchmarks saturate

The authors motivate the benchmark with a paired experiment on 150 WebCompass-edit tasks, holding the rubric, judge and render pipeline fixed:

  • Qwen3.7-Plus single-shot (one response with search/replace blocks): 73.6% mean graded completion, 32.7% full pass.
  • The same model agentic (inspect, edit, run, iterate): 95.1%, 69.3% full pass.
  • Claude Opus 4.8, agentic: 99.4%.

A 21.5-point gain just from letting the model run and check its own work. When a benchmark's tasks are simple enough that a self-verifying loop solves almost all of them, scores stop separating models. (The paper notes the cross-benchmark comparison uses different task pools and protocols, so read Co-Design Bench's lower numbers as "harder, real tasks", not a direct regression.)

Takeaway: if your internal eval was built for single-shot prompting, it may be near-saturated for an agentic setup. Refresh the set with the requests your users actually struggled with, the same trick the authors used: mine rephrasings, repeated follow-ups and rollbacks from your logs.

Lesson 4: more tokens didn't buy more completion

On multimodal edits, GPT-6 Astra used a median of 32 turns and 1.24M tokens per case for 0.754 strict completion. GLM-5.2 used 69 turns and 7.29M tokens for 0.572. On repository edits, Qwen3.8-27B used 6.72M tokens for 0.645, versus 2.33M for GPT-6 Astra at 0.667. The authors note token traffic includes retries and cache usage, so it's an execution measure, not a common dollar cost.

Takeaway: log tokens and turns per completed task. A model that loops longer isn't trying harder in a useful way, and your bill pays for every turn.

How they graded, and what you can copy

There's no gold code and no target screenshot. Instead the judge-then-refute protocol:

  1. Build and render the edited app in a sandbox, collect source diffs, DOM, screenshots and Playwright traces.
  2. An agentic judge grades each hidden criterion Pass / Partial / Fail from evidence: a screenshot for visual claims, recorded browser actions for interaction claims. A handler existing in source code doesn't prove a user can reach it.
  3. A separate LLM verifier challenges every Pass and Partial, looking for stubs, hardcoded outputs, unreachable entry points, failures under varied inputs and requirement mismatches.
  4. Missing evidence aborts scoring rather than falling back to a weaker proxy.

Two frontend experts checked rubric scores on 200 task–model outputs, and re-scoring two models with three alternative judge models moved graded scores by at most 0.053 without changing their relative order.

This is a pattern you can borrow for your own agent evals: make the grader prove each pass with evidence, then have a second pass try to refute it. Agents are good at producing things that look done.

Limits worth stating

  • Each task is a fixed request: no clarifying questions, no mid-task corrections. Real pair-programming has a human in the loop, so these numbers measure autonomous completion.
  • Grading is by LLM judges, which can be wrong (the authors say so).
  • The ten models were the authors' choice, the task set skews toward hard, previously failed requests, and almost all repository tasks are single-page React apps (171 of 173).
  • Small rank differences, like Qwen3.8-27B edging Opus 5 on repository edits (0.645 vs 0.642), are not evidence of a general advantage, as the paper itself notes.

So don't quote "agents fail 3 of 4 tasks" about everyday coding. The honest reading: on hard, real revision requests, even the best model–harness pair usually leaves at least one requirement unfinished.

A checklist for evaluating your own coding agent

  • Headline metric = task success: every required criterion passes. Keep per-criterion coverage as the diagnostic.
  • Write acceptance criteria per task, hidden from the agent, including "don't break X" preservation checks.
  • Grade from evidence (tests, screenshots, recorded interactions), and add a refute pass that hunts for stubs and hardcoded results.
  • Evaluate the model + harness pair you'll ship; re-run evals when either changes.
  • Source tasks from real failures: rephrased requests, repeated follow-ups and rollbacks in your logs.
  • Track tokens and turns per completed task, not per attempt.
  • Break results down by task type (multi-file vs local edits); averages hide where the agent breaks.

The gap between "mostly right" and "done" is where your users live. Measure that gap directly.

How does your team measure whether an AI coding agent actually finished the job?

References