What does mutation testing reveal that coverage does not, and where is it worth the cost?
Show the full answer Hide the answer
What it reveals
Whether tests actually assert anything.
Coverage measures which lines were executed. A test that calls a function and asserts nothing produces full coverage and catches nothing — and coverage-target-driven suites contain a great deal of exactly that.
Mutation testing changes the code deliberately — flipping a comparison, altering a constant, removing a call — and checks whether any test fails. A surviving mutant means no test detected the change, which means the behaviour is not actually verified.
It measures assertion quality, which is the property that matters and which coverage does not measure at all.
Where it is worth the cost
In the code where correctness matters most and defects are expensive:
- Pricing, billing and financial calculation.
- Permission and authorisation decisions.
- Data integrity invariants.
- Algorithms with subtle boundaries.
- Anything where a defect is silent rather than loud.
Where it is not
- Across an entire codebase, which is prohibitively slow — the analysis runs the test suite many times over.
- On code dominated by integration and wiring, where mutations are either uninteresting or produce equivalent programs.
- On generated code or thin adapters.
Making the cost manageable
- Scope to changed code, running only mutations in the diff — which fits the pipeline and targets where attention is warranted.
- Limit the mutation operators to the ones that produce meaningful findings.
- Run it periodically on critical modules rather than continuously everywhere.
- Treat surviving mutants as findings to review, not as a score to maximise — some mutants are equivalent and unkillable, and chasing them wastes effort.
The framing
Coverage tells you what was executed; mutation testing tells you what was verified. Used narrowly on high-consequence code, it is the most direct available answer to "do these tests actually work?" — a question coverage cannot answer and that most suites would fail.