Language models generate texts that sound plausible. This is their strength and at the same time their greatest risk: a result can be linguistically flawless and factually entirely fabricated. In technical terminology, this behaviour is called hallucination – the model fills gaps with invented but convincingly formulated information instead of admitting ignorance.

A current court case demonstrates where this leads without control. There, both parties relied on pleadings in which judgments were cited that do not exist. The lawyers involved were sanctioned for this. The time saved that the tools had brought in drafting the documents turned into reputational and liability damage.

Why this does not only affect lawyers

The mechanism is transferable. Anyone who uses AI systems in project and content workflows creates content in many places that looks finished at first glance:

  • Code: Generated functions call methods that do not exist in the library version used, or follow patterns that would be correct in a different framework.
  • Technical documentation: Configuration parameters, paths or commands are invented because they fit the style of the real documentation.
  • Editorial content: Figures, studies, standards or reference customers appear in the text without existing.
  • Legal and regulatory statements: Deadlines, paragraphs or obligations are summarised without checking whether the source exists and is current.

The difference from human error lies in the error culture of the interface. An overwhelmed employee formulates uncertainly, leaves gaps or asks questions. A language model delivers a closed, confident answer even when the basis is missing. It is precisely this sovereignty that lowers the willingness of the people who process the text further to check it.

Responsibility remains with the organisation

The decisive point of the court case is not the technology but the attribution. It was not the tools that were sanctioned but the persons who submitted their results unchecked. This principle also applies in the corporate context: anyone who delivers a quotation, documentation, a privacy policy or a piece of production code is liable for the content – regardless of which aid contributed to its creation.

This results in an uncomfortable calculation. The time saving of an AI tool is only real if the review time is included in the calculation. If the review is omitted in order to maximise the efficiency gain, the effort is merely shifted into the future – in the form of rework, bug fixing or, in the worse case, in the form of disputes with customers.

What a robust review process must deliver

A sustainable approach to generated content does not need new bureaucracy but a few clear rules that are also adhered to in everyday work:

  1. Label the origin. It must be traceable which parts of a document or repository were created by machine. Only then can the review effort be managed in a targeted way.
  2. Obligation to provide evidence for facts. Every verifiable claim – figure, standard, reference, library function – is checked against a primary source. Invented information almost always stands out on the first search for the original passage.
  3. Four-eyes principle for outgoing documents. Everything that leaves the house or goes into production is approved by a second person who can assess the context.
  4. Automated counter-checking in code. Static analysis, type checking, linting and tests catch a large part of invented calls before a human even reads them.
  5. Form risk classes. An internal meeting transcript needs less control than a contract text or a payment module. The review effort should be aligned with the possible damage, not with gut feeling.
  6. Binding usage rules. A short internal policy clarifies which tasks AI may be used for, which data may be entered and who approves.

Culture beats tool

Technically, the hallucination risk can be dampened, for example by having models work with verified internal sources instead of answering from memory. It can be reduced this way, but not eliminated. That is why the working culture is decisive: teams need explicit permission to discard generated results, and they must not be measured against time targets that are only achievable without review.

It is also helpful to make errors visible. Anyone who briefly documents invented information that was noticed in review builds up, over months, a realistic picture of where the tools used work reliably and where they do not. That is a better basis for decisions than any general debate about trustworthiness.

What this means for web and software projects

For projects, this means: quality assurance is not an appendage that can be saved with AI support, but the part that gains weight through it. Anyone who uses generated content in editorial work, documentation and development should plan and budget for review as a fixed work step.

The honest efficiency gain only becomes apparent after deducting this review time – and even then it is still considerable.

Sources