Quality and Testing Strategy

Definition

A quality strategy allocates verification effort across levels — unit, integration, contract, end-to-end, production — to maximise confidence per unit of cost, where cost includes runtime, maintenance, flakiness and the delay imposed on every change.

The economics

As verification moves up the levels it becomes slower, flakier, more expensive to maintain and harder to diagnose. So each level should test what the level below cannot, and the level above should be rationed.

The common inversion — few unit tests, many end-to-end tests — produces a suite that takes 40 minutes, fails randomly, and is eventually bypassed. Once a suite is routinely ignored it provides no confidence at any cost.

The properties that matter more than the shape

Determinism. A flaky test is worse than no test: it trains the team to re-run rather than investigate, and eventually masks a real failure. Quarantine flaky tests immediately; fix or delete within a window. Deleting is legitimate.

Speed. A suite over about ten minutes changes behaviour — people batch changes and stop running it locally.

Failure clarity. A failing test should say what broke. A test asserting forty things says only that something is wrong.

Testing behaviour, not implementation. Tests coupled to internal structure make refactoring expensive, which is the opposite of what tests are for.

Where confidence comes from beyond tests

Some confidence is only obtainable in production, and pretending otherwise produces enormous pre-production suites that still miss production behaviour:

• Canary releases with automated comparison.
• Synthetic transactions exercising the full path including clients and third parties.
• Feature-flagged rollout to a cohort.
• Observability good enough to detect a problem quickly, which substitutes for some pre-production verification.

The quality measures worth tracking

Change failure rate, time to restore, escaped defect rate, and — the one that matters most for the test suite itself — how often it catches a real defect that would otherwise have reached production. If that is rarely, the suite is ceremony regardless of its coverage.

Interview question

"How would you decide whether your test suite is worth what it costs?"