Autonomous AI agents have long been regarded as the next step after the chat window: systems that not only respond but break down tasks, call tools, check intermediate results and ultimately deliver a finished work product. A research experiment dampens this expectation. Researchers had AI agents write scientific studies on previously unanswered research questions – the programs failed at the task.

The setting was deliberately chosen to be hard. An open research question is precisely the case in which the answer cannot be assembled from existing material. There is no pattern in the training material that a model could orient itself by, and no authority that incidentally confirms the result. Anyone who wants to succeed here must clearly delineate their own research question, choose a method, recognise contradictions in their own approach and subsequently be able to discard their own work as inadequate. That is a different demand from generating a plausibly sounding text.

Plausible is not the same as correct

The interesting point for companies is not that agents make mistakes. Humans do that too. It lies in where the mistakes occur and how difficult they are to recognise. Language models are optimised to produce results that correspond to the form of correct results. A study text that looks like a study is therefore achievable. A study text whose insight holds up is not automatically so.

Transferred to software projects, this means: an agent will with high probability deliver code that compiles, follows a convention and looks reasonable in structure. Whether it solves the actual domain problem, whether it takes into account an edge case that nobody wrote down, and whether it will still be maintainable in three years, is decided at a level that the model cannot evaluate as long as it receives no genuine feedback from reality.

Self-improvement needs an external corrective

The hope behind self-improving agents is: the system checks its own result, recognises the weak point and iterates until it fits. In practice, this presupposes that the checking is harder than the generation. That is exactly where it breaks down. If the same model is author and reviewer, it checks against the same assumptions that already led to the error.

Where this corrective exists mechanically, agents work significantly better. A failing test, a linter error, an HTTP status 500, an unmet database condition – these are unambiguous signals from outside that an agent can work through. Where the corrective is missing or only a human can supply it, autonomy ends.

What this means for practical use

A useful selection rule for projects can be derived from this distinction. Agents make sense where three conditions come together:

  • Verifiable result: There is an automated test, a schema, a validation or a reference against which the result fails or passes.
  • Limited damage: A failure costs computing time and a repetition, not data integrity, trust or money.
  • Clear stopping point: The agent cannot run on endlessly but has defined limits for attempts, runtime and access.

Tasks in which the evaluation itself is the actual achievement, by contrast, remain critical. A code review that places architectural decisions in context. The question of whether a requirement should be implemented in this way at all. The interpretation of a contractual or regulatory requirement. In these cases, an agent is a proposer and not a decision-maker.

Human control points belong in the workflow, not alongside it

The point at which the human intervenes is important. An approval at the end of a long agent chain is of little value because checking the overall result there is more labour-intensive than creating it oneself would have been. Control points at the transitions are more useful: before an agent receives write access to a system, before it executes a migration step, before content is published.

Transparency about the path to the result is equally helpful. If it remains traceable which tools an agent called and which intermediate steps it discarded, an error can be localised. Without this trail, the only choice left is between blind trust and complete post-checking.

Expectation management is part of project work

The finding from the research experiment is not an argument against AI in everyday development. It merely specifies what the technology is currently good for. Assistance with clearly defined, verifiable work steps – there is solid experience for that. Independent handling of open questions without specialist supervision – there is not.

For web and software projects, this above all implies a planning task: agents should be introduced with the same requirements for testability, permission allocation and logging as any other automated component. Those who build the checking loops first and expand autonomy afterwards gain speed without incalculable risks. And those who quantify the benefit realistically will not later have to retract an expectation that was never technically covered.

Sources