How would you decide whether your test suite is worth what it costs?

What is being tested

Whether you can evaluate verification effort as a portfolio rather than assuming more is better.

The measure that matters

How often does it catch a real defect that would otherwise have reached production?

If the answer is rarely, the suite is ceremony regardless of its coverage percentage — and coverage is a proxy that is frequently gamed by tests asserting the code does what it does.

Track it: for each escaped defect, ask which level should have caught it. For each caught defect, note which did. Over a quarter that produces a real picture of where verification is earning its cost.

The costs to count

Not just runtime:

• Maintenance — tests updated on every refactor, which is the largest hidden cost.
• Flakiness, which costs re-runs and, far worse, trains the team to ignore failures.
• Delay imposed on every change, multiplied by change frequency.
• False confidence, where a green suite conceals an untested path.

The signals that the balance is wrong

Inverted pyramid — few unit tests, many end-to-end. Slow, flaky, and eventually bypassed. Once a suite is routinely ignored it provides no confidence at any cost.

High coverage, frequent escapes — testing the wrong things.

Tests coupled to implementation, making refactoring expensive, which is the opposite of what tests are for.

A suite over ten minutes, which changes behaviour: people batch changes and stop running it locally.

The redistribution when it is wrong

Push logic coverage down to unit tests. Add contract tests for service interactions — the level most often missing, and what replaces most end-to-end coverage in a distributed system. Keep three to five end-to-end journeys. And move some confidence to production: canary analysis, synthetic transactions, flagged rollout.

Some things are only observable in production, and enormous pre-production suites that try to substitute for that still miss them.

The complementary measures

Change failure rate, time to restore, and escaped defect rate. If change failure rate is low and the suite is slow and flaky, the suite is over-invested. If change failure rate is high and coverage is high, it is testing the wrong things.