AI agents – that is, systems that independently plan and execute tasks based on a language model – have long ceased to be a laboratory topic. They answer customer enquiries, search documents, generate code or trigger processes in connected systems. With this spread, interest is growing among those who want to misuse such systems.
Security researchers have investigated how fraudsters use AI agents in attacks. One finding is remarkably sobering: despite modern technology, the human factor behind the methods remains recognisable. The tools change, the patterns behind them less so.
Why this is relevant for companies
For decision-makers, this observation is more important than it initially sounds. If attack methods continue to bear a human signature, protection concepts cannot be based on technical filters alone. Organisation, responsibilities and processes matter just as much.
At the same time, the attack surface is shifting. A classic web project has clearly defined entry points: forms, interfaces, uploads. An agent, by contrast, processes texts from very different sources – emails, tickets, websites, files – and can derive actions from them. Each of these sources is potentially an input that influences the behaviour of the system.
Prompt-based attacks made understandable
A prompt is the instruction used to control a language model. Prompt-based attacks aim to take over precisely this control: a text that the agent reads as part of its work contains instructions that it interprets as a legitimate task.
The problem lies in the architecture of many systems: data and instructions end up in the same text channel. A model does not reliably distinguish between "this is content I should summarise" and "this is a command I should follow". The more rights an agent has in connected systems, the greater the potential damage.
In addition: as soon as several models or agents work together, complexity increases. Outputs from one system become inputs for the next. Faulty or manipulated content can thus run through a process chain without anything looking obviously wrong at any single point.
Guardrails instead of a leap of faith
Anyone deploying agents productively should treat them like any other executing component – with defined rights, logging and checkpoints. An approach that combines technology and organisation has proven effective:
- Minimise rights: An agent only gets access to the systems and data sets that it actually needs for its specific task.
- Safeguard critical actions: Writing or irreversible operations – payments, deletions, publications, permission changes – go through human approval.
- Treat inputs as untrusted: External content is data, not commands. This separation must be reflected in the system design, not just in the prompt formulation.
- Establish traceability: Which inputs did the agent process, which decision followed from them, which tool did it call? Without a log, later analysis is hardly possible.
- Test the limits: Agent behaviour should be systematically tested, including with deliberately manipulated inputs, before the system goes live.
The human factor remains
The observation that human patterns still lie behind the attacks points to a second lever: awareness. Employees who work with AI-supported tools should understand that a plausibly formulated output is not proof of quality. An agent can sound convincing and still follow someone else's instruction.
In practice, this means: clear rules on when a result is cross-checked, which data may even enter an AI system and to whom unusual system behaviour is reported. These rules should be documented, not passed on verbally.
Classification for web and software projects
For web and software projects, part of the security work thus shifts from classic input validation to the design of agent logic: which rights, which data sources, which approval points. Anyone integrating AI functions into a CMS, a customer process or a development environment should answer these questions before the first productive use – not afterwards. An agent with narrowly defined rights and clean logging remains manageable; an agent with broad rights and no checkpoints becomes a risk that is difficult to reconstruct in an emergency.