Websites are no longer read only by search engines and humans. AI crawlers — automated programs that collect content for training data or for the answers of AI assistants — access the same pages. Two recent reports show that this has become a technical contest of strength: one from the side of the operators defending themselves against scraping, one from the side of an AI provider that is putting a stop to manipulation attempts.

Shieldfont: inconspicuous for humans, unusable for machines

A designer duo has released a font called Shieldfont that is intended to disrupt the collection of data by AI crawlers. The principle: for visitors, nothing changes in the display; the page looks as it did before. What the crawler extracts from the source code, however, no longer produces meaningful text.

Technically, such an approach operates on well-known terrain: a font maps character codes to specific images of letters. If this mapping deviates from the standard, the visible result no longer matches what is actually stored in the document as a character string. That is precisely the lever — and precisely where the risk lies.

The price: accessibility and findability

Anyone who deliberately falsifies machine-readable text does not only affect AI crawlers. All systems that depend on the same text are affected:

  • Screen readers that read out the content work with the character stream in the document, not with the rendered image.
  • Copy and paste: copied text ends up in the clipboard in the falsified form.
  • Search engines and internal search, which index content and also read the raw text in the process.
  • Translation and read-aloud functions in browsers as well as reader modes.

For companies that must meet legal requirements for digital accessibility, this is not a marginal note. A measure that locks out machines in practice also locks out assistive technologies — and for part of the user base, these are the only means of access. Such techniques are therefore at most suitable for clearly delimited areas in which both accessibility and visibility in search engines are deliberately set aside. For content that is meant to be found and read, they are not a tool.

Perplexity: hidden advertising in Markdown variants

The second report turns the picture around. Here it is not the operator protecting itself, but the AI provider defending against manipulation. According to its own account, Perplexity is taking action against advertising that is hidden in special Markdown versions of websites — that is, in lean, machine-readable text variants that providers increasingly offer in addition to the normal HTML page. Markdown is a simple text format with minimal formatting; such variants are easier for crawlers to process than a complete website with layout, scripts and navigation.

Precisely this separation opens a gap. What is in the Markdown variant is not seen by any human visitor. Anyone who places advertising messages or recommendations there addresses exclusively the AI system — with the aim of appearing in its answers. Perplexity assesses this as a manipulative tactic and sees users' trust in search results at risk, because paid content could appear in an answer without being recognisable as advertising.

Two sides of the same problem

Both cases are based on the same cause: the version of a website that humans see and the version that machines read are no longer necessarily identical. This divergence can be exploited in both directions — defensively, to supply crawlers with false content, or offensively, to feed them additional content.

From the perspective of search engines, this is an old acquaintance. Cloaking, that is, serving different content to crawlers and to users, has been considered a violation of search providers' guidelines for years. That an AI provider is now enforcing comparable rules was foreseeable. For operators, this means: techniques based on a divergence between visible and machine-readable content are not a stable foundation. They are detected and sanctioned as soon as they occur at scale.

What operators can do instead

The legitimate question behind Shieldfont remains: how much of one's own content should flow into AI systems? There are ways to do this that damage neither accessibility nor trust:

  • Access control via robots.txt and crawler identifiers: many AI crawlers identify themselves and respect exclusion rules. This is not a guarantee, but it is the documented, traceable path.
  • Server-side control: rate limits and the evaluation of access patterns in the logs show which bots read and to what extent — the basis for any further decision.
  • Content behind login: what really should not flow out does not belong in the publicly retrievable part of the website.
  • Deliberate handling of machine-readable variants: anyone offering Markdown or API versions should treat them editorially like the main page. Content parity here is a compliance question, not a question of taste.
  • Label advertising as advertising — in every output variant, including the one that only machines read.

For web and software projects, this means above all one thing: the machine-readable version of a website is no longer a by-product, but a delivery format of its own with its own requirements for quality and governance.

Anyone who provides additional variants for AI systems needs the same editorial control for them as for the main page and must be able to technically ensure that the content is identical.And anyone who wants to limit access by AI crawlers should regulate this at the access layer — not through tricks in rendering that ultimately affect their own users.

Sources