The advertising message is tempting: a language model with a context window of one million tokens is supposed to process entire projects in one go. You upload the repository, ask a question and receive an answer that takes the entire context into account. A recent analysis on Golem.de points out that this is precisely what does not work reliably in practice: despite advertised context lengths of millions of characters, the models fail with large documents.
For companies that want to integrate AI into development, documentation or editorial work, this is a relevant limitation. Because it shifts the effort: it is not the larger model that solves the problem, but the clean preparation of the inputs.
What the context window actually is
A context window refers to the amount of text that a model can take into account at the same time in a single request. It is measured in tokens, i.e. in text building blocks that correspond to roughly a word fragment depending on the language. Everything that is in this window is visible to the model: the system instruction, the history of the conversation, attached files and the actual question.
However, the size of the window only says how much fits in. It says nothing about how well the model actually uses the individual passages within it. This is precisely where the gap between technical specification and output quality lies.
Lots of context does not mean lots of attention
The longer the input, the more the information it contains competes for the model's limited processing capacity. With extensive documents and codebases, this leads to individual details being lost or incorrectly linked. The model then does not respond with a clear indication of missing information, but formulates plausible-sounding statements that are not accurate in detail.
With code, this effect is particularly unpleasant. A class that is defined in one place and used in fifty others requires precise references. If the model slightly shifts a method signature or invents a configuration option, this is hardly noticeable when reading, but leads to errors in implementation. The review effort increases precisely where the time saving was promised.
Costs and latency grow too
Alongside quality, economic efficiency plays a role. Billing is usually per token, including for the input. Anyone who sends a complete project with every question pays again with every iteration for material that is largely irrelevant to the specific task. In addition, the response time increases, which quickly becomes disruptive in interactive tools in everyday development.
The seemingly simplest way, to throw everything in, is therefore also the most expensive and the slowest. And it does not deliver the best result.
What works instead
The alternative consists of giving the model less, but better-selected material. A few approaches have become established for this:
- Targeted selection instead of full submission. Only put into the context the files and sections that are actually needed for the task: the affected class, its direct dependencies, the associated tests.
- Sensible segmentation. Split large documents into sections that are complete in terms of content, instead of cutting them by character count. Half a paragraph or a split function are worthless as context.
- Retrieval on demand. In so-called retrieval procedures, suitable passages of text are searched for in an index based on the question and only these are included. The model then works with a few relevant excerpts instead of the entire stock.
- Condensed intermediate levels. Architecture overviews, module descriptions or maintained interface documentation are more compact than the source code and often more informative for questions at system level.
- Break down tasks. Several clearly defined requests with a narrowly scoped context deliver more reliable results than one large request that is supposed to clarify everything at once.
Documentation quality becomes a technical factor
This leads to an uncomfortable but useful insight: the effectiveness of AI tools depends on the order of one's own material. A project with clear module boundaries, descriptive names and up-to-date documentation can be worked on with a model much better than a historically grown system without structure.
The same applies to content processes. Anyone who keeps editorial content, product data or guidelines in a maintained, structured form can feed them into requests in a targeted way. Anyone who instead resorts to scattered files in different formats merely shifts the problem into the model's context, where it is not solved but only becomes invisible.
What to check when introducing
When selecting tools, it is therefore worth looking not at the advertised context length, but at the question of how a system decides which material it presents to the model in the first place. Tools that make transparent which files and sections they have included are considerably more useful in troubleshooting than those that conceal the process.
Equally sensible is one's own small series of tests with typical questions from one's own project. Whether a model performs well on a text example from the advertising brochure says little about how it behaves in the concrete legacy system.
Classification for web and software projects
Large context windows are a useful technical advance, but they do not replace information architecture.
In web and software projects, it therefore pays to do the work that makes sense anyway: clear module boundaries, up-to-date documentation, structured content.These very foundations determine whether AI support saves time in everyday work or creates additional review effort.