Anyone who lets an AI agent work over an extended period eventually hits a hard limit: the context window. This refers to the amount of text that a language model can oversee at once. When it is exhausted, the conversation history so far is summarised and passed on in shortened form. This process is called context compression and is the prerequisite for agents to be able to stay on a task across many steps at all.

Research by scientists at Penn State now shows that in the process, what is systematically lost is precisely what must be lost least: the conditions that users themselves have set. On average, 83 per cent of these specifications do not survive compression. A typical example is the instruction not to send an email without explicit approval.

Why rules in particular disappear

A summary follows a content-based logic: it preserves what appears to be the core of the task. But a secondary condition such as a mandatory approval is precisely not that — it is a marginal note that was formulated once at the beginning and then no longer appears in the conversation history. It is exactly this structure that makes it the first candidate for trimming.

In practice, this is an unpleasant combination. The agent does not lose its ability to act, only the restriction of that ability. It keeps working, appears competent — and in doing so does something that it had been forbidden to do. An error that does not present itself as an error is considerably harder to detect in operation than a crash.

A small module as memory for conditions

As a countermeasure, the research group proposes a specialised add-on module based on a comparatively small model with around nine billion parameters. Its task is narrowly defined: it is to recognise user conditions from the conversation history and ensure that they survive compression. In the trials, this preserves over 90 per cent of the restrictions.

What is remarkable about this approach is less the technology than the division. It is not the large model that is made better, but a specific weak point that gets its own small component. This is a pattern that has proven itself in software architecture for decades: a critical function is outsourced instead of being co-managed in a general-purpose building block.

Prompt instructions are not a guardrail

For companies integrating AI agents into business processes, this yields an uncomfortable realisation. Very many rules in such systems today sit in the prompt — that is, in the instruction text that the model receives at the start. Specifications formulated there feel binding, but technically they are not. They are text among text and compete with everything else for the scarce space in the context.

A rule only becomes binding when it is enforced outside the model. In practice, this means:

  • Secure critical actions technically. A send, a booking, a deletion should not be prevented by the model remembering a prohibition, but by the executing interface doing nothing without confirmed approval.
  • Attach permissions to the tool, not the prompt. Which functions an agent may call at all belongs in the configuration of the tools and in the permission layer — that is where it can be checked and logged.
  • Keep conditions in a structured form. User specifications should be maintained as a separate, machine-readable list and re-injected at every step, instead of standing once in the conversation history.
  • Treat long sessions as a risk factor. The longer an agent works on a task, the more often compression has occurred. Sensible cut-offs, intermediate handovers and limited runtimes reduce the number of compression steps.
  • Test for rule loss. Acceptance testing should include the question of whether a condition set at the beginning still applies after many steps — not just whether the specialist task is solved.

Consequences for approval logic

The finding is particularly relevant for all processes with a four-eyes principle. When an agent prepares quotes, posts content into a content management system or drafts customer correspondence, approval is the actual control point. If this control point lies solely in the model's memory, then it is not present, only probable.

The clean solution is trivial and yet often skipped: the agent creates a draft with a defined status, and a separate step not controlled by the model raises this status. This is classic workflow modelling, as has long been customary in editorial and approval processes. The AI component sits before the control point, not above it.

Assessment

The figures from the study are an indication of a class of errors that is only indirectly related to model quality: it arises from the interplay of memory limits and summarisation logic. Whether an add-on module of this kind will become widely available is open — the architectural lesson applies regardless.

For web and software projects, this above all means a shift in design. Everything that must not happen belongs in permissions, status models and interfaces, not in instruction texts to a language model.

Whoever integrates agents in this way gains twice over: the process remains auditable, and those involved do not have to trust the model in order to be able to hand work over to it.

Sources