AI coding agents are becoming increasingly capable of writing software, modifying applications and automating development workflows. However, a new benchmark suggests that they still struggle to complete complex programming assignments without human intervention. In a test based on real-world software development requests, the best-performing AI model fully completed just 26.4% of tasks, leaving at least one requirement unfinished in the remaining assignments.
The benchmark, called Co-Design Bench, was developed by researchers from Alibaba and several Asian universities. It evaluates whether AI agents can understand user instructions, modify existing web applications and deliver working results. The findings highlight a gap between AI systems that can complete most individual requirements and those that can reliably deliver an entire software change. They also raise questions about how coding-agent performance should be measured as businesses increasingly adopt AI-assisted development tools.
Key Takeaways
- The best-performing AI coding model fully completed only 26.4% of tasks in Co-Design Bench.
- The leading model satisfied approximately 69.9% of task requirements on average.
- Researchers tested 10 AI model endpoints across 273 software-development tasks.
- The benchmark was built from 37,463 deduplicated requests across 26,314 real coding sessions.
- The tasks included modifying existing web applications using text, images, audio, video and documents.
- Multi-file repository changes were more difficult than localised application edits.
- Every tested model performed worse when tasks combined three or four types of changes.
- The study suggests that human review remains important when AI agents modify existing software.
New Benchmark Tests AI Coding Agents on Real User Requests
The researchers created Co-Design Bench to address a growing limitation in traditional AI coding benchmarks.
Many existing tests focus on relatively isolated programming problems, such as fixing a specific bug or implementing a clearly defined feature. These tasks can be easier for AI systems to solve because their requirements are narrow and the expected outcome is relatively straightforward.
Real software development is different.
A developer might ask an AI agent to update a dashboard, change its design, modify how data is calculated and ensure that several related pages display the updated information correctly.
Completing only part of that request does not necessarily produce a usable result.
Co-Design Bench attempts to measure this broader challenge by testing agents on requests taken from actual coding sessions.
The research team began with 37,463 deduplicated user requests collected from 26,314 real coding sessions. It eventually selected 273 tasks for the benchmark.
Each task was paired with a working application and hidden acceptance criteria written by experts.
These criteria allowed researchers to evaluate whether the requested changes had actually been implemented rather than simply checking whether the AI had generated code.
What Does Co-Design Bench Include?
The benchmark contains two main categories of software-development tasks.
| Benchmark track | Number of tasks | What it tests |
|---|---|---|
| Multimodal Editing | 100 | Localised application changes using requests that can include images, audio, video and documents |
| Repository Editing | 173 | Coordinated changes across multiple files and application components |
| Total | 273 | Real-world application modification |
The benchmark includes 2,331 hidden acceptance criteria.
These criteria examine three broad capabilities:
- Grounded understanding: Whether the AI understands the user’s instructions and supporting information.
- User-interface transformation: Whether it makes the requested design and layout changes.
- Functional implementation: Whether the modified application actually works as intended.
The tasks range from small adjustments, such as extracting a price from a receipt and updating related totals, to more extensive application changes involving multiple components.
Best AI Coding Model Fully Solves Only 26.4% of Tasks
The most important result is the difference between partial completion and full task success.
GPT-6 Astra ranked first among the 10 tested model endpoints. It satisfied approximately 69.9% of the acceptance criteria on average.
However, it fully satisfied every requirement in only 26.4% of tasks.
This means the model often completed a substantial portion of the requested work but missed at least one requirement needed for a complete result.
The distinction matters because a software application can appear to have been successfully modified while still containing an unfinished feature, an incorrect interaction or an inconsistency between different parts of the application.
For example, an AI agent might update a dashboard’s headline figure but fail to update the underlying table or other components that depend on the same information.
The application may look correct at first glance, but the overall task remains incomplete.
How Did Other AI Models Perform?
The research compared 10 model endpoints using a shared agentic development environment.
GPT-6 Astra achieved the highest full-task completion rate, followed by Claude Opus 5 and Qwen3.8-Max.
| AI model | Fully completed tasks |
|---|---|
| GPT-6 Astra | 26.4% |
| Claude Opus 5 | 18.3% |
| Qwen3.8-Max | 16.5% |
| Kimi K3 | 15.8% |
| Qwen3.8-27B | 15.0% |
| GPT-5.6 Sol | 14.3% |
| Claude Opus 4.8 | 10.3% |
| Grok 4.6 | 6.6% |
The results show that even the highest-ranked model struggled to meet every requirement.
These percentages measure complete task success, not the proportion of code generated or the average number of individual requirements satisfied.
They also apply specifically to the benchmark’s selected tasks and testing environment. They should not be interpreted as the success rates of these models across every programming assignment.
Complex Software Changes Are More Difficult for AI Agents
One of the clearest patterns in the research was that performance declined when a task required several types of changes at the same time.
A simple request might involve changing a heading or adjusting the layout of a page.
A more complex request could require changing the layout, updating content, modifying interactions and ensuring that the underlying application logic remains consistent.
These requirements are connected. Making one change without correctly handling the others can leave the application in an incomplete state.
The researchers found that all 10 tested model endpoints performed worse when repository-editing tasks involved three or four types of changes instead of one or two.
This is particularly important because real software development frequently involves interconnected components.
Why Multi-File Changes Create Problems
Modern applications rarely operate as isolated pages.
A single feature can depend on several components, including:
- Front-end design and layout
- Application logic
- Database structures
- Data validation
- User interactions
- Navigation and routing
- Connected APIs and external services
An AI agent may correctly modify one component but fail to update another component that depends on it.
For instance, changing the way a shopping application calculates discounts may also require updates to the product page, shopping cart, checkout process and order summary.
If one of these components continues using the old calculation, the application can produce inconsistent results.
This helps explain why completing individual coding requirements is easier than delivering a fully functioning application update.
Researchers Use Browser Testing to Check Whether Code Actually Works
Co-Design Bench also attempts to address another weakness in traditional coding evaluations: generated code can look convincing without implementing the requested functionality.
The researchers developed a grading approach called judge-then-refute.
The first AI evaluator checks each acceptance criterion using evidence from the source code and the running application in a browser.
A second AI verifier then attempts to challenge positive evaluations and identify cases where the apparent implementation is incomplete.
The process also includes additional checks for visual and interactive claims.
For example, an agent should not receive credit simply because it creates a static mock-up of a working feature. The feature must function in the application.
This approach is designed to make evaluation stricter and reduce the risk of rewarding superficial implementations.
However, the researchers acknowledge that AI-based evaluators can still make mistakes.
The Coding Environment Can Change AI Performance
The study also highlights the importance of the software environment used to operate coding agents.
A coding agent does not work through its underlying AI model alone. It also relies on a harness, the software system that allows it to read files, edit code, execute commands, inspect results and interact with the application.
In a separate experiment, researchers ran the same models using different harness configurations.
They found that performance could change significantly depending on the environment.
For example, Qwen3.8-27B achieved different results on larger repository overhauls when tested through different harnesses.
The relative ranking of models could also change depending on the environment.
This suggests that comparing coding agents only by their underlying model names can provide an incomplete picture.
A company’s choice of development environment, tools and testing workflow can influence how effectively an AI coding agent performs.
For businesses selecting coding assistants, evaluating the model and its operating environment together may provide a more useful assessment.
Why Traditional AI Coding Benchmarks May No Longer Be Enough
The researchers argue that some established coding benchmarks have become increasingly saturated.
When AI agents can generate code, execute tests and automatically correct mistakes, they can achieve extremely high scores on older evaluation tasks.
The result is that these benchmarks may no longer distinguish reliably between stronger and weaker coding systems.
Co-Design Bench takes a different approach by using real requests that previously created difficulties during software development.
It also evaluates whether the entire requested change works rather than rewarding partial implementation.
This does not mean older benchmarks have become useless. They can still measure important programming capabilities.
However, the new research suggests that evaluating AI coding systems requires more realistic tasks that test understanding, coordination, implementation and verification together.
What the Findings Mean for Developers and Businesses
The findings have practical implications for companies increasingly using AI agents to build and maintain software.
AI coding tools can help developers generate code, investigate bugs, update existing applications and automate repetitive work.
However, businesses should be careful about treating generated code as production-ready without testing it.
The benchmark suggests several important considerations.
Human code review remains necessary: Developers should inspect changes, especially when an AI agent modifies several connected components.
Tasks should have clear acceptance criteria: Breaking a large request into smaller, testable requirements makes it easier to identify incomplete work.
Automated testing matters: Browser tests, integration tests and regression checks can help identify problems that are not obvious from generated code.
The development environment matters: Companies should evaluate the complete coding-agent setup instead of comparing models only through headline benchmark scores.
Complex changes need additional supervision: Tasks involving design, data, interactions and application logic together deserve more careful review.
The broader lesson is that AI coding agents can accelerate development, but speed of code generation should not be confused with successful software delivery.
The Bigger Picture
The Co-Design Bench findings provide a useful reality check for the growing AI coding industry. AI agents are becoming capable of performing increasingly complicated development work, but completing most of a request is not the same as delivering a fully functioning result. The gap between 69.9% average requirement completion and 26.4% full-task success demonstrates why software quality must be measured through working applications and complete acceptance criteria.
For businesses and developers, the likely direction is not an immediate replacement of human programmers but a shift in how software is built. AI agents can handle substantial portions of development, while people remain responsible for defining requirements, reviewing changes and verifying the final product. As models and development environments improve, the key measure of progress will be how consistently these systems can deliver complete, reliable software rather than simply generate more code.
Frequently Asked Questions
1. What is Co-Design Bench?
Co-Design Bench is an AI coding-agent benchmark developed by researchers from Alibaba and several Asian universities. It evaluates whether AI systems can modify existing web applications based on real user requests.
2. How many tasks did researchers test?
The benchmark includes 273 tasks selected from 37,463 deduplicated requests across 26,314 real coding sessions.
3. Which AI model performed best?
GPT-6 Astra ranked first among the 10 tested model endpoints, fully completing 26.4% of tasks and satisfying approximately 69.9% of requirements on average.
4. Why do AI coding agents struggle with complex tasks?
Complex tasks often require coordinated changes across application design, content, interactions, data and multiple code files. An agent can complete some changes while missing other connected requirements.
5. Does the study prove that AI cannot replace programmers?
No. The research evaluates a specific set of difficult application-modification tasks. It does not establish that AI agents cannot perform other programming activities or that human developers will always be required for every coding task.
6. What is a coding-agent harness?
A harness is the software environment that allows an AI coding agent to access files, edit code, execute commands, run tests and inspect application behaviour.
7. Why are traditional coding benchmarks becoming less useful?
Some established benchmarks have become easier for advanced agents, which can generate, test and repair their own code. This can produce very high scores that make it difficult to distinguish between models.
8. What should developers do when using AI coding agents?
Developers should define clear requirements, run automated tests, review generated code and verify that every requested feature works correctly before accepting changes.
Looking Ahead
AI coding agents are likely to become more capable as models improve and development environments provide better tools for testing, debugging and coordinating changes across applications. The Co-Design Bench research suggests that progress will need to be measured through complete task delivery, not simply the amount of code an AI system can generate or the percentage of individual requirements it satisfies.
For software companies, the opportunity is to combine AI’s speed with human engineering judgment. Agents can take on repetitive implementation work and help developers move faster, but reliable software still requires clear specifications, testing and quality control. The next generation of coding tools will need to demonstrate that they can consistently turn complex user requests into complete, working applications.
Get the day’s top stories in your inbox
One concise email. No spam, unsubscribe anytime.
