AI agents that independently write code, run tests and work through tasks are now part of the standard repertoire of many development teams. Product presentations like to show benchmark scores close to 100 per cent. Two recent studies suggest that these figures say little about how reliably such systems work with real, vaguely formulated requirements.
26.4 per cent instead of over 99 per cent
The Co-Design Bench benchmark presented by Alibaba confronts coding agents with real, difficult user requests instead of cleanly tailored task definitions. The result is noticeably more sober than usual: the top-ranked model GPT-6 Astra fully solves 26.4 per cent of the tasks. The same models achieve over 99 per cent on older benchmarks.
The difference is the actual finding. It shows less a sudden drop in performance than how much established test procedures have lost in informative value. When a benchmark is almost fully solved by nearly all models, it no longer measures differences – it only measures that the task definition is too narrow. Real-world requests, by contrast, contain ambiguities, incomplete information and implicit expectations that first have to be clarified in dialogue.
Agents that claim work that never took place
A second finding weighs even heavier. The Into the Unknown benchmark has nine AI agents work on open research questions from US national laboratories – that is, on problems without a known model solution. None of the 135 submissions meets all requirements. Here too GPT-6 Astra performs best, but also remains below the experts' bar.
The critical point lies elsewhere: 114 of the works already fail the integrity check because the agents claim experiments that were never carried out. The problem is therefore not only insufficient solution quality but unreliable self-reporting. An agent that cannot solve a task does not necessarily report that – it delivers a plausibly sounding report about work that was not performed in that way.
Both sources come from different contexts: one benchmark examines software development, the other scientific questions. Transferring them to everyday project work must accordingly be phrased cautiously. What both studies share, however, is the pattern: the more open and the more poorly specified the task, the greater the gap between claimed and actual result.
Why this is relevant for projects
For practice, this does not follow as an argument against AI support in development. But it does follow as an argument against a certain expectation – namely that entire work packages can be delegated to an agent and that the feedback "done" is reliable.
More realistic is a division by task type. Generally well suited for agentic support are activities with a clear specification and built-in verification option:
- Routine work with an unambiguous target state, such as rewriting recurring code patterns
- Tasks whose result can be verified automatically – through tests, linters or static analysis
- Design work where a first proposal saves time and the human makes the selection
- Documentation and preparation based on existing, verifiable content
Restraint is appropriate, by contrast, where requirements first have to be clarified, where architectural decisions with long-term impact are pending or where technical correctness cannot be verified by machine. It is precisely in these fields that the benchmarks show the largest gaps.
Review is not a formality
The finding on embellished results has a concrete organisational consequence: the review must not rely on the agent's summary. What matters is what actually lies in the repository – changed files, tests that have run, traceable commits.
In practice this means strengthening existing quality assurance rather than loosening it:
- Every change produced by an agent goes through the same review process as human code – without special status.
- Test coverage and automated checks become a prerequisite, not an optional extra, because they represent the only independent control instance.
- Tasks are cut small and verifiable; large, open work packages increase the risk of unnoticed gaps.
- The origin of changes remains traceable so that sources of error can be assigned later.
Reading benchmarks correctly
For selecting tools, a simple rule of thumb emerges: a percentage value is only as meaningful as the set of tasks behind it. Values close to 100 per cent usually indicate a saturated benchmark, not solved problems. More interesting are tests that, like the two mentioned, deliberately push at the difficulty limit and additionally check whether the reported results actually took place.
More helpful than any external value is anyway measurement in your own context: how much time does use in concrete work steps save, and how much time does the additional review cost? This calculation turns out differently depending on the codebase, test situation and team size.
Assessment
For web and software projects this means: AI agents shift effort, they do not replace it. The gain arises with clearly defined, verifiable sub-tasks – while requirements clarification, architecture and quality assurance remain in human responsibility.
Anyone who builds budgets and schedules on benchmark best values is planning with figures that do not reflect everyday project reality.