Anyone who operates AI agents productively knows the basic problem: with every additional step, the conversation history grows, and with it the amount of data that is sent to the model again with every request.Anthropic is now introducing so-called Message Threads for its Messages API.Instead of sending the entire conversation again with every turn — that is, with every individual exchange between application and model — agents now only transmit the new messages.

According to Anthropic, this reduces the time to first token by around 35 percent for requests with 500,000 to one million tokens.Tokens are the text units into which a language model breaks down inputs; one million tokens correspond to a very extensive context, such as arises in long research or code editing runs.

Why the repetition was so expensive

Classic chat APIs are stateless.The model remembers nothing between two calls; the complete history must therefore be supplied with every call.For short dialogues this is unproblematic.However, for agents that call tools over hours, collect intermediate results and process documents, the history grows continuously.

The result is a kind of silent friction loss: every additional step becomes slower than the previous one, even though the actual task remains equally complex.For users, this manifests as longer pauses before any output appears at all — precisely the value that Anthropic addresses with Message Threads.

Incremental instead of complete

The approach shifts the management of the conversation history to the provider's side.The application only sends the increment, and the thread maintains the context.Technically, this is not a new model and not a new capability of the system, but an optimisation of the transport path.That is precisely why the change is interesting for project managers: it promises a measurable improvement without any change to the logic of the agent itself.

An effect on costs is obvious, because less data is moved.However, in the present report only the latency gain for very large requests is documented in a robust way.Anyone who wants to calculate with concrete savings should measure this for their own use case instead of adopting a rule of thumb.

Which applications this is relevant for

The stated range of 500,000 to one million tokens clearly marks where the benefit is greatest.Typical candidates are:

  • Agents that call tools over many steps, such as database queries, search services or internal APIs
  • Processes with extensive document context, for example contract, quotation or support documents
  • Development assistants that keep larger codebases in context
  • Dialogue applications with long session duration, in which the history does not end after a few minutes

For short, clearly delimited requests — such as a one-off classification or a text summary — the difference is hardly significant by contrast.There, the bottleneck is usually elsewhere.

What to consider during implementation

A conversation managed on the server side changes the architecture of the application.Until now, the complete history was usually held in the application's own system or database, and the API was a pure computation step.If the thread is held at the provider, a state arises outside the application's own system.

This gives rise to several practical questions that every team should answer for itself:

  1. Data storage: Which content will be located where in future, and how does that fit with existing data protection and deletion concepts?
  2. Error cases: How does the agent behave if a thread is no longer available or a step has to be repeated?
  3. Traceability: Can the history still be fully represented in the application's own logging, for example for audits or error analyses?
  4. Portability: How much effort would a switch to another provider involve if a provider-specific thread concept is used?

None of these questions argues against using it.However, they determine whether an optimisation later becomes a dependency or remains a controlled component.In practice, it has proven worthwhile to maintain the complete history in parallel in the application's own system as long as the effort is acceptable.

Classification within a larger trend

Message Threads are part of a development that has been emerging for some time: competition is shifting from pure model capabilities to operational capability.It is no longer only the question of whether a model can solve a task that decides on productive use, but whether it does so quickly, reliably and at calculable cost.

Latency in particular is often underestimated.An application that delivers professionally excellent results but noticeably stalls every time will be bypassed in everyday use.Improvements in time to first token therefore contribute directly to acceptance.

What this means for web and software projects

For projects in which agents have so far failed due to response times or operating costs, the feasibility threshold shifts a little.Use cases with long context that previously seemed too sluggish thus move into the realistic range.

At the same time, the following applies: such optimisations are infrastructure decisions and should be taken into account early in the architecture, rather than being retrofitted into an existing integration.

Sources