For decades, robots.txt has been the simplest control file on the web: a text file in the root directory of a domain in which operators specify which automated programs may retrieve which areas. With the rise of AI crawlers – programs that collect content not for search but for training language models – this file has become confusing. There is a separate identifier for almost every provider, which must be entered individually.

Google wants to change that. According to consistent reports, the company is pushing an cross-provider token that is to be standardised in the IETF. The IETF (Internet Engineering Task Force) is the body in which fundamental internet standards are agreed – the Robots Exclusion Protocol, i.e. robots.txt itself, was also formally described there. From 2027, Google intends to evaluate such a shared token.

One entry instead of many individual rules

The practical core of the change is quickly explained: instead of maintaining a separate rule for each AI provider, a single entry should suffice to prohibit or permit the use of one's own content for AI training. This reduces maintenance effort and lowers the error rate, because every manually maintained list becomes outdated the moment a new provider comes onto the market.

According to the available information, the existing Google-specific identifier Google-Extended will be retained. In future it will serve for exceptions – i.e. for cases where operators want to set a different rule for Google than for the rest of the market. This creates a two-tier model: a general policy decision via the new token, with finer-grained rules per provider below it if needed.

The well-known fundamental problem: voluntariness

The second source sets a different emphasis and points to the weak point of the procedure: robots.txt is simply ignored by many AI crawlers. It is not a technical access protection but a declaration of intent. Anyone who does not comply is not technically prevented – the file is publicly readable on the server and contains only instructions whose compliance is voluntary.

That is precisely why there is discussion about making the procedure more binding. A standard adopted in the IETF initially creates clarity about how an opt-out is to be expressed in a technically correct way. What enforcement power actually results from this and how many providers will join cannot be derived from the available information. Put cautiously: a uniform token eliminates the ambiguity on the part of website operators, but does not automatically eliminate the behaviour of uncooperative crawlers.

What operators can sensibly prepare now

Even without fixed details, it is worth clarifying one's own starting position. The following points are useful regardless of the specific token name:

  • Stocktaking: Which robots.txt files exist at all – per domain, per subdomain, per staging environment? In grown setups, there are often more than those responsible suspect.
  • Clarify responsibility: Who in the company decides whether content is released for AI training? This question is strategic and not purely technical – it touches marketing, legal and management.
  • Establish maintainability: robots.txt should not be set in stone as a static file in the deployment, but should remain changeable via the CMS or a versioned configuration. In TYPO3, the delivery of the file can be solved configurationally instead of placing it manually on the server.
  • Monitoring of accesses: Server logs show which bots are actually active. Without this data basis, any discussion about crawler control remains theoretical.
  • Make a distinction: Search engine crawling and AI training are two different things. A blanket blocking can cost visibility if the rules are formulated too coarsely.

Timeline

2027 is named as the point in time for Google's evaluation of the new token. Until then, today's provider-specific entries remain the only way to express preferences. Anyone who has already set an opt-out should leave it in place and add to it later rather than switching prematurely.

Assessment for web and software projects

For ongoing projects, this above all means: robots.txt is turning from a technical side issue into an artefact with business relevance, about which a conscious decision should be made. Anyone who sets up the file today in such a way that it can be changed without deployment and is clearly documented can add a future standard token within a few minutes. And anyone who also evaluates the actual bot accesses in parallel makes this decision on the basis of data rather than assumptions.

Sources