Assertion Strength

Coverage measures whether a line executed during a test, not whether the test would fail if the line were wrong. A test that calls a function and asserts nothing produces full coverage and zero protection, and weak assertions are extremely common — particularly in tests written to raise a coverage number.

High coverage with defects in tested code is therefore the expected outcome rather than an anomaly.

Why it matters

Coverage is the most widely used quality proxy and it is measuring the wrong property. An organisation optimising it will produce tests that execute code without checking it, and the number will improve while the protection does not.

Implementation patterns

• Mutation testing: introduce small changes — invert a condition, change a boundary, remove a statement — and check whether any test fails. A surviving mutation is a line that is executed and unprotected, and each survivor names a specific missing assertion.
• Scope it, because it is computationally expensive. Run it on the modules where correctness is expensive — calculation logic, state machines, money paths — rather than everywhere. A high mutation score on a premium calculation matters; on the request-parsing layer it does not.
• On a schedule rather than per commit, and on changed code in the pipeline where the cost permits.
• The cheaper approximation: review the assertions in the critical modules' tests by hand, once. A surprising proportion assert only that no exception was thrown, and finding them takes an afternoon rather than a compute budget.
• Test behaviour rather than implementation, so the suite fails on regressions and not on refactoring — which is a different failure mode with the same symptom of a suite nobody trusts.

Industry example

Insurance platforms such as Digit and Acko have exactly the code where this matters most: premium calculation, eligibility rules and claims adjudication are where a wrong result is silent — the arithmetic works, the number is wrong, and nobody notices until a regulator or a customer does.

Those modules are usually the best-covered by line coverage, which is precisely why coverage is misleading there — the tests exist and their assertions are weak.

Failure scenarios

• Coverage as the quality target, producing executed-but-unchecked code.
• Mutation testing applied to the whole codebase, making it too slow to run.
• Tests asserting only that no exception was thrown.
• Tests asserting implementation details, failing on refactoring and passing on regressions.
• The mutation score treated as a target, which produces the same gaming as coverage did.

Trade-offs

Mutation testing is expensive in compute and produces findings that take time to act on, and a low score on a module can be legitimate if the module genuinely has little worth asserting.

The value is in the specific survivors rather than in the score, which is why it is better used as a diagnostic on chosen modules than as an organisation-wide metric — a metric would be gamed exactly as coverage is.

Interview question

"You have 90% coverage and a defect shipped in a covered line. Explain how that is possible, and tell me what you would measure instead."