In many projects, the answer to the question about the test status is a number: this many test cases, this many of them automated. This number is easy to collect and looks reassuring in status reports. However, it says little about the actual reliability of the software. This is precisely where a current discussion in the testing community comes in: Richard Seidl spoke with Ivaylo Brüssow and Andrej Nod about a Metric Collector – a tool that collects metrics on test automation and evaluates the result as a score.

The basic idea is simply stated but uncomfortable in practice: whoever counts test cases measures effort. Whoever collects metrics measures impact. The difference becomes relevant where teams need to know whether they can trust their test automation in a critical situation.

Why the number of test cases is misleading

A high number of test cases can mean three very different things. It can stand for well-thought-out coverage. But it can also be the result of redundancy, where the same logic is checked again and again in slightly modified form. And it can simply be legacy: tests that no one maintains anymore, that regularly turn red and whose failures the team ignores out of habit.

None of these three cases can be distinguished from the pure count. For decision-makers, this means: a metric that looks identical in all three scenarios is not suitable as a management measure. It suggests control without delivering it.

What a Metric Collector is supposed to achieve

A Metric Collector – that is, a component that automatically gathers measurements from repositories, test runs and build pipelines – pursues a different approach. Instead of a single number, a bundle of metrics is created, which is then condensed into an assessment. The score is not an end in itself but a means of communication: it makes the state of test automation comparable across teams and projects and shows changes over time.

The automation of the collection itself is crucial. Metrics that someone has to maintain manually in a spreadsheet quickly become outdated and eventually stop being maintained at all. Values that are generated during every build anyway, by contrast, remain available permanently and without additional effort.

What practically matters with metrics

Which specific measures make sense depends on the project. However, some guardrails have proven themselves in project practice when teams switch from counting to assessment:

  • Stability before quantity. Tests that sporadically fail without code changes undermine trust in the entire suite. Their frequency is a more meaningful measure than the total number.
  • Runtime and feedback time. A test suite whose result is only available after hours changes behaviour in the team – changes are bundled, causes of errors are harder to assign.
  • Distribution across test levels. Whether the focus is on fast unit tests or on slow, maintenance-intensive UI tests significantly influences costs and informative value.
  • Coverage of business-critical paths. Not every line of code is equally important. What is interesting is whether the business-critical processes are safeguarded.
  • Maintenance state. Deactivated or permanently skipped tests are a warning signal that is completely lost in pure counts.

A score summarises such dimensions. It does not replace the technical discussion, it structures it: if the value falls, it is possible to look specifically at which dimension has deteriorated.

The connection with AI-supported development

The question of robust test metrics is gaining weight because code is increasingly created with the support of language models. Generated code can be produced very quickly – including generated tests. This shifts the bottleneck: it is not the writing of tests that is the scarce resource, but the assessment of whether these tests actually safeguard anything.

When quantity hardly costs any effort anymore, quantity finally loses its informative value as a quality indicator. A suite can grow to several hundred test cases within a short time without a single additional error being found. In this situation, objective metrics are not a luxury but the prerequisite for being able to make any statement about reliability at all.

What metrics cannot do

For all their usefulness, there is one limitation that teams should plan for early: as soon as a metric becomes a target, it can be optimised without quality improving. A coverage rate can be raised with tests that have no meaningful assertions. Flaky tests can be removed by deactivating them instead of repairing them.

A score should therefore be understood as an indicator that triggers conversations – not as a target against which teams are measured. The benefit lies in development over time and in comparison between systems, less in the absolute value of a single month.

Getting started without a big programme

A switch to metric-based assessment does not have to begin as an initiative with its own project budget. As a rule, a pragmatic start is enough: define a handful of metrics, integrate their collection automatically into the existing pipeline and observe the values over a few releases. Only when the data basis is stable does it pay off to condense them into a score.

It is important that the metrics come from the same source as the builds themselves.

Only then are they low in manipulation and remain up to date, even when something else is more urgent in everyday project work.

Classification for project practice

For web and software projects, the relevant question in status meetings shifts: no longer how many tests exist, but how stable, how fast and how targeted they are. Anyone who collects these metrics automatically gains a basis for discussion that carries across team boundaries and makes investments in quality assurance comprehensible. Especially where AI tools accelerate code production, the informative value of the test base determines whether this pace leads to reliable releases or to additional rework.

Sources