End-to-end tests fail intermittently and nobody owns them. QA says the developers broke them; developers say the tests are flaky. How do you resolve this?
Show the full answer Hide the answer
The ownership gap is the actual problem
A test suite owned by nobody is maintained by nobody, and each failure becomes a negotiation rather than a fix. That is the condition to change; the flakiness is downstream of it.
Get the data before adjudicating
For the last hundred runs: which tests failed, how often, and did the same commit both pass and fail? That single question separates the two claims — a commit that passes and fails without changing is flakiness, not a broken feature — and it settles the argument with evidence rather than by seniority.
Expect both parties to be partly right, which is the usual outcome and is worth saying out loud early.
Assign ownership to the team that owns the behaviour
Not to QA as a function. The team that owns the checkout service owns the checkout end-to-end test, because they are the only ones who can tell a genuine regression from a timing problem, and because a test they own is a test they will keep green.
QA's role shifts to the framework, the environment and the practice, which is where their leverage actually is.
Then apply the economics
A suite failing intermittently is probably too large. At ninety-nine percent per-test reliability, a two-hundred-test suite fails spuriously more often than it passes.
Classify by what each test uniquely proves: keep the journeys whose failure would be a serious incident, move business rules and edge cases down to unit and integration level, move service compatibility to contract tests, and delete duplicates.
Thirty reliable end-to-end tests provide more actual coverage than two hundred unreliable ones, because the thirty are trusted and acted on.
Quarantine the remainder with a cap and a deadline
Detect flakiness automatically, move it out of the blocking suite, raise an owned ticket, and delete on expiry. And investigate causes rather than assuming — a meaningful proportion of flaky tests are correctly detecting real non-determinism in the system, and those are the most valuable findings in the whole exercise.