How should the number of end-to-end tests be decided, and what is the honest accounting?
Show the full answer Hide the answer
The honest accounting
Cost per end-to-end test is far higher than it appears, and most of it is ongoing:
- Execution time, multiplied by every run, contributing to a slow pipeline that changes developer behaviour.
- Infrastructure for a full environment.
- Maintenance, since they break on any change to any component they touch.
- Flakiness triage, which is the largest hidden cost — and a suite with known-flaky tests trains everyone to re-run rather than investigate, which destroys the value of every reliable test.
- Diagnosis time, since a failure says something is broken somewhere.
Value per test is also lower than it appears, because most defects they catch would have been caught by a cheaper layer — a unit test, a component test or a contract test.
What only they catch
The residual value is real and narrow: integration assumptions no contract expresses — ordering, timing, configuration, data-state dependencies, and the interaction of components that are each individually correct.
That is a small set, which is why the layer should be small.
How to decide the number
Start from critical user journeys, and keep the list short. Sign-up, sign-in, the primary transaction, checkout. Not features — journeys, defined by what a user is trying to accomplish.
Then apply the test: would a failure here be a serious incident? If not, the journey does not need an end-to-end test; a component or contract test suffices.
Most organisations converge on a number in the low tens, and organisations with hundreds have accumulated them rather than decided them.
The rule that keeps it honest
Every end-to-end test needs an owner and a justification, reviewed periodically. Without that, they accumulate — each added after an incident, none removed — until the suite is slow, flaky and ignored.
And a flaky end-to-end test should be deleted, not tolerated. It provides no assurance and actively degrades the assurance the rest of the suite provides.