Benchmarks for AI models often measure things that have little to do with everyday project work: multiple-choice questions, coding puzzles, exam tasks. The Remote Labor Index takes a different approach. It puts AI systems to work on real freelance assignments, i.e. work packages that real clients have actually commissioned and paid for, and assesses whether the result meets professional standards.

The current figures are remarkable. According to this measurement, Anthropic's model Fable 5.2 completes 41.3 per cent of assignments to professional quality. The next-placed systems Opus 5.5 and GPT 6 Astra reach less than half that value. To put the pace in context: a year earlier, the best measured value was still 2.5 per cent.

Why this benchmark is more relevant than others

For companies commissioning web or software projects, the difference between a test task and commissioned work is decisive. A real assignment brings with it everything that clean benchmarks lack: incomplete briefings, contradictory requirements, file formats that nobody has documented, and sign-off by someone who has a concrete result in mind.

If a model under these conditions solves four out of ten assignments in such a way that the result passes as professional, that is a different statement from a high score in an exam format. At the same time, the flip side applies: six out of ten assignments simply do not reach this level. The benchmark does not describe a replacement of skilled workers, but a shift in the threshold at which automation pays off.

Where the models still fall short

Product design and animation are explicitly named as weak points. That fits with what can be observed in projects. Both disciplines are defined less by clear targets than by weighing up options: which variant carries the brand? Which movement feels fluid, which feels hectic? What can still wait and be extended in six months?

Such decisions are hard to derive from a briefing because the briefing does not contain them at all. They emerge in conversation, in comparing several drafts and in experience of which solution comes back to haunt you later. That is precisely where the contribution of humans remains irreplaceable so far.

What can sensibly be accelerated in projects

A practical rule of thumb can be derived from the results: the more clearly a task is outlined, the more verifiable its result and the tighter its context, the more likely AI support will help. Typical candidates in web and TYPO3 projects include:

  • Data preparation before a migration, i.e. cleaning, restructuring and mapping existing content
  • Creating and maintaining technical documentation from existing code and existing configurations
  • Routine work in development such as test coverage, recurring refactorings or adjustments to changed interfaces
  • Text variants, metadata and translation drafts that are subsequently reviewed editorially
  • Research and comparison tasks whose result can be verified against a source

One should be more cautious where design, brand management, information architecture or long-term system architecture are involved. Not because models deliver nothing there, but because the cost of a wrong decision only becomes visible late – and is then expensive.

Checking becomes the actual work

A success rate of around 40 per cent has an organisational consequence that is easily overlooked: someone has to determine which 40 per cent it is. Without reliable sign-off, the effort is merely shifted rather than reduced. Anyone using AI in a project therefore needs defined review steps, clear responsibilities and task scopes in which a result can be assessed at all.

That argues for small, clearly delimited work packages instead of large assignments to a model. The shorter the distance between assignment and control, the lower the risk that an error carries through subsequent work steps.

Plan for the pace, don't wait

The jump from 2.5 to 41.3 per cent within a year should be read less as a forecast than as an indication of planning uncertainty. Nobody can seriously say whether this curve will continue. For project decisions, however, it is relevant that the situation can change noticeably within a typical project duration.

In practice this means: build processes and systems so that individual work steps can be swapped out without the overall system depending on them. Clean interfaces, documented workflows and well-maintained data are the prerequisite for this – and they pay off regardless of how the models continue to develop.

Assessment for web and software projects

For web and software projects, what shifts above all is the shape of the work: clearly outlined, verifiable tasks can be completed much faster today, while design, architecture and weighing up options continue to require experience. Those who reallocate budgets accordingly – less routine, more conception and quality assurance – gain speed without risking project quality. What remains decisive is that every AI-supported intermediate output has a defined review step before it flows into a productive system.

Sources