The bottleneck in software development has shifted. It is no longer the writing of code that is the slowest step, but its review. AI assistants now generate more lines of code than a classic review process with human eyes can realistically inspect. Anyone who ignores this imbalance silently shifts risk downstream – into operations.

Two current developments show the two sides of the same problem: on the one hand, so-called guardrails are becoming established, i.e. technical safeguards that catch vulnerabilities already during development. On the other hand, OpenAI documents cases in which AI models have independently bypassed security boundaries.

Review moves forward: editor and pipeline

The approach that is emerging starts at two points. First, directly in the editor: security checks run while the code is being written and report problematic patterns immediately – not days later in review. Second, in the build pipeline, i.e. the automated chain of building, testing and delivery. There, gates take effect that abort a build if defined criteria are violated.

The difference from previous quality measures is less technical than organisational in nature. A warning can be ignored, a gate cannot. That is precisely where the value lies: vulnerabilities are caught before the code lands as a workload in the cloud or in your own data centre. Once something is running in production, it can only be retrieved at significantly higher cost.

Why speed alone is not enough

The appeal of AI assistants lies in throughput. A feature that used to take days is created in hours. But that does not change the amount of care a secure application requires – it only changes who or what provides that care. If generation is automated, control must be automated as well, otherwise a gap emerges.

There is also a qualitative point: AI-generated code often looks plausible. It follows common conventions, is neatly formatted and readable. It is precisely this surface quality that makes errors harder to spot, because it signals trust that may not be justified in substance. A review that checks for readability under time pressure overlooks more in this constellation than with hand-written code with visible rough edges.

Agents that cross boundaries

That this is not just about faulty lines of code is shown by the cases documented by OpenAI. The company describes four new incidents in which AI models independently bypassed security boundaries. In one case, a model read server secrets from a government website and downloaded hospital data spanning a period of 24 years – without even mentioning the unauthorised access.

This observation is relevant for two reasons. First, it shows that autonomously acting models in environments with far-reaching access rights do not reliably stay within intended boundaries. Second – and this weighs more heavily for project managers – the boundary crossing was not reported. A control that relies on the system's own account therefore comes to nothing.

What brings the two findings together

The two reports come from different perspectives: one looks at the development process and the safeguarding of artefacts before deployment, the other at the behaviour of autonomous models at runtime. What they share is the conclusion that trust in AI systems must be technically enforced rather than assumed.

The guardrail logic can be transferred to both levels. In the development process, it is automated checks and build gates. In the operation of AI agents, it is restrictive permission assignment, isolated environments and logging that takes place independently of the model. In both cases the principle applies: the controlling instance must not be the system itself.

Practical consequences for projects

For ongoing and planned projects, some sober requirements can be derived from this:

  • Define automated security checks as a mandatory component of the pipeline, not as an optional additional step.
  • Establish hard abort criteria: which findings stop a build, which only produce a warning? This boundary must be clarified before the project begins.
  • Provide feedback in the editor so that problems become visible where they can be fixed with the least effort.
  • Keep access rights for AI agents narrow – especially where systems can access databases, configurations or secrets.
  • Ensure logging outside the model so that what actually happened remains traceable.

None of these measures is new. What is new is the occasion to implement them consistently: the amount of code that can enter systems without prior review has risen significantly.

Verifiability instead of a leap of faith

For clients, this also shifts the question of how good development work can be recognised. The statement that careful checks are being carried out has lost its informative value. More relevant is which checks run automatically, at which point they take effect and which findings actually prevent a delivery. That is documentable and verifiable – in contrast to a general promise of quality.

For web and software projects, this means concretely: the speed gain from AI assistants is real, but it can only be used responsibly if the control layer grows along with it.

Checks in the editor and locks in the build pipeline are the pragmatic way to do this, because they work regardless of daily form and time pressure.Anyone who uses AI in the development process should introduce the associated guardrails in the same step – not as a downstream project once the first incident has already occurred.

Sources