In many companies, a simple rule of thumb has become established: the most powerful available model is used for each AI-supported task. The logic behind this sounds plausible – whoever takes the best tool gets the best result. In practice, however, this attitude regularly leads to budget being spent on computing power that would not have been necessary for the specific task at all.

A current specialist article puts the point into a catchy formula: counting tokens is not enough. What matters is what a completed task costs – not how many resources were consumed in the process.

What token efficiency actually measures

Tokens are the smallest processing units of a language model – simplified, word components by which inputs and outputs are billed. Because a clean number can be derived from this, token consumption is readily used as a control variable.

The problem: this number says nothing about whether the result was usable. An inexpensive model that has to be revised three times and ultimately still requires manual correction is more expensive than a stronger model that delivers a usable result on the first pass. Conversely, a premium model for a simple classification task is simply oversized.

The more meaningful metric is therefore cost per completed task. In addition to the pure model costs, this also includes:

  • repeat runs because the first result was not sufficient
  • effort for human review and post-processing
  • loss of time due to slower responses in interactive workflows
  • follow-up costs when faulty outputs are noticed only late

Model routing as an architectural decision

From this perspective, model selection becomes an architectural question. Instead of using a single standard configuration for everything, each task class is assigned to the model that solves it reliably and at justifiable cost. This approach is referred to as model routing: an upstream logic decides which request goes to which model.

In practice, typical tiers can be distinguished:

  1. Structured routine tasks such as categorisation, extraction of fields from documents or simple reformulations. Smaller, fast models are usually sufficient here.
  2. Tasks of medium complexity such as summaries, drafts for texts or code fragments in manageable context.
  3. Tasks with high requirements for traceability, such as refactorings across multiple files, architectural decisions or the analysis of contradictory requirements. Here a strong model pays off.

The economic effect arises because the large majority of requests typically fall into the first two categories – while the cost per request is highest in the third category.

What this means for coding workflows

The difference becomes particularly clear in AI-supported software development. An assistant that suggests completions in the editor is called extremely frequently. Here, low latency and low cost per call matter more than maximum model strength. An agent that, by contrast, prepares a larger change across multiple modules benefits from a model with better planning capability – because a failed run is expensive here, both in computing time and in developer time.

Anyone who covers both cases with the same configuration either pays too much or gets results that are too weak. A deliberate separation is therefore not an optimisation detail, but a fundamental decision in the setup.

Context is a cost factor

A second lever lies in how much information is given to a model. Every additional context – entire files, long chat histories, complete documentation – increases token consumption on every single call. Often it does not improve the result proportionally, but even dilutes it, because relevant passages are lost in the mass.

Practical starting points are therefore:

  • targeted selection of the context excerpts actually needed instead of blanket handover
  • clearly formulated task descriptions that reduce follow-up questions and repetitions
  • caching recurring components, insofar as the platform used supports this
  • limiting output length where compact answers are sufficient

Making it measurable instead of estimating

A routing strategy can only be evaluated if results are measurable. For this, each use case requires an idea of what an acceptable result is – for example a test suite for code, a sample check for text generation or defined mandatory fields for data extraction.

Only on this basis can a smaller model be pitted against the previous one on a trial basis. If the hit rate remains stable, the switch is a pure cost advantage. If it falls, it shows where the additional model strength is actually needed. This review is not a one-off process: since the model offering is constantly changing, regular re-evaluation is worthwhile.

Organisational prerequisite

Technically, model routing is manageable. The organisational side is usually more difficult. When several teams in a company connect AI services independently, parallel configurations quickly arise without a shared cost logic. A central abstraction layer through which requests run creates transparency here – and makes it possible to swap models without touching every application individually.

Equally important is an allocation of costs to concrete use cases.

A total bill without a breakdown does not allow any decision about where optimisation is worthwhile.

Context for web and software projects

For projects in which AI functions run permanently in operation, model selection becomes a fixed component of the technical design – comparable with caching strategies or the choice of database.Anyone who provides for an interchangeable interface from the outset and defines per use case what constitutes a good result keeps both costs and quality under control.The decisive shift in perspective lies in not declaring the most powerful available technology to be the norm, but rather the respectively sufficient one.

Sources