The question of which AI model belongs in a project was quickly answered a few years ago: there was hardly any choice. Today, even individual providers maintain entire model families that differ in size, response behaviour, response time and price. For development and automation projects, this is not a pleasant abundance but a decision that must be justified.
In practice, it is often made on gut feeling: you take the model you know from the chat window, or the one that is currently being written about the most. Both are poor selection criteria because they do not take the actual task into account. A model that convinces in free dialogue can disappoint with strictly structured data extraction – and vice versa.
First the task, then the model
The first step is unspectacular but is often skipped: describing the task precisely. Not "summarise texts", but: which input, which length, which language, which output format, which errors are tolerable and which are not.
This description usually already yields a shortlist. Typical task classes in web and software projects are:
- Classification and routing: sorting enquiries, assigning tickets, tagging content. Here consistency and speed count more than linguistic elegance.
- Extraction: obtaining structured data from documents. What matters is that the output format is reliably adhered to.
- Text production: drafts, translations, variants. Here differences in quality have an immediately visible effect.
- Multi-step tasks: research, code changes, workflows with tool calls. Such cases place the highest demands and often justify a more expensive model.
This classification has a practical side effect: it shows that in a project, one model is rarely the right answer. The simple pre-sorting of inputs and the demanding final processing can be served by different models.
Making quality measurable
Without your own measurement, every model choice remains a matter of taste. What is needed is not a scientific procedure but a small test set of real cases from the project – preferably including the unpleasant exceptions that cause problems in everyday work.
Each case requires an idea of what an acceptable result looks like. With structured outputs, this can be checked automatically; with texts, a short rating scale and a second pair of eyes help. What matters is repeatability: the same test set is run again with every model change.
An often underestimated point is variance. Models do not respond deterministically; the same input can produce different outputs. A model that is good on average but unpredictable may be less suitable in an automated process than a somewhat weaker but more stable one.
Calculating costs realistically
Billing is typically based on tokens, i.e. text and data units for input and output. The price per unit alone says little. What is meaningful is the amount per completed transaction – per document checked, per enquiry answered, per draft generated.
It is also worth looking at volumes: an expensive model in a process that runs ten times a day is uncritical. The same model in a process with tens of thousands of runs significantly changes the project calculation. Add to this the indirect costs that are easily overlooked: rework by humans, additional review steps, repetitions for faulty outputs.
What counts besides quality and price
Some criteria decide the choice faster than any quality comparison because they set hard framework conditions:
- Response time: In interactive interfaces, waiting time is a quality feature. In nightly batch runs, it is irrelevant.
- Data protection and operating location: Where is data processed, what may the company hand over at all, is operation in its own infrastructure necessary.
- Context length: How much material must the model oversee in one pass.
- Availability and lifespan: Models are replaced. Anyone who pins an application to exactly one model plans for migration effort without knowing it.
Interchangeability as an architectural decision
Because the offering is constantly changing, the most important decision is often not the one for a model but the one for a structure in which models can be swapped. In practice, this means: bundling model access behind your own interface, storing prompts and evaluation criteria in a versioned way, logging results.
Once this is in place, a strategic fundamental question becomes a routine task: connect the new model, run the test set, offset the costs per transaction, decide. This cycle takes hours instead of weeks.
What this means for projects
Model diversity does not require deeper expertise in model architectures but project discipline: a clear task description, your own test set, cost calculation per transaction. Anyone who has these three things can objectively assess every new launch instead of following it or ignoring it.
And anyone who builds interchangeability into the architecture from the outset keeps open the possibility of working more cheaply or better in twelve months – without touching half the system.