AI coding helpers fully finish only about 1 in 4 real tasks

AI coding helpers are not as finished as their old scores suggest. In a new test built from real user requests, the best of ten models, GPT-6 Astra, fully completed only 26.4 percent of tasks. On older tests, the same kinds of models score above 99 percent.

What the test found

Researchers from Alibaba and several Asian universities built Co-Design Bench. Agents must rework existing web apps based on user requests. Some requests include images, audio, video, or documents. GPT-6 Astra met about 70 percent of each task's requirements on average. But it usually missed at least one detail, so the result was fully correct only about one time in four.

Claude Opus 5 and Qwen3.8-Max followed GPT-6 Astra. Grok 4.6 came last.

Why older tests missed it

The researchers say older benchmarks have become too easy. Models can run, check, and fix their own code, so scores bunch near the top. On one older test, WebCompass-edit, the gap to the new test was more than 35 percentage points.

Co-Design Bench uses real requests from coding sessions. The team favored cases where users had to reword a request several times or undo changes. Tasks range from small tweaks, like pulling an amount from a receipt and updating every total that depends on it, to full project overhauls.

Grading is strict. One AI judge tests the finished app in a browser. A second re-checks every pass and looks for fake mock-ups or hardcoded answers. Points go only to features that truly work.

Where the agents struggled

• Scores fell for every model once a task mixed three or four types of change, such as layout, design, and content.
• GPT-6 Astra fully solved 41 percent of tasks with images, audio, or video, and just under 18 percent of larger overhauls.
• GLM-5.2 used nearly six times as many tokens, a rough measure of computing effort, as GPT-6 Astra on smaller changes and still scored worse.

The setup matters too

The team also ran the same models in different harnesses. A harness is the program an agent uses to read, write, and run code. Qwen3.8-27B scored 75 percent in one setup and 56 percent in another on larger overhauls. The authors suggest reporting the model and its setup together. Another recent study, however, found barely any benefit from more elaborate harnesses.

What to keep in mind

The test has limits. Each task is one fixed request, with no follow-up questions or mid-task corrections. The AI judges can also be wrong. Separate research by METR found project leads accepted AI-written code about 24 percentage points less often than scores on the popular SWE-bench Verified test suggested.

For a business, the practical point is simple. Plan for a person to check and finish AI-built software work. Try the tool, and the setup around it, on your own tasks before relying on a leaderboard.