beginner 2 min answer

Two tests exercise the same line of code. One starts the process and drives the HTTP API; the other calls the function directly. Once you look at the failure signal rather than the code covered, what actually distinguishes them?

test-pyramidfeedback-loopdiagnosiscietsy
Show the full answer Hide the answer

What the layers really measure

Not how much code they touch. The difference is the set of causes consistent with a failure, which is worth naming: the diagnostic radius.

When the direct call fails, the cause is in one function or in the test. When the HTTP test fails, the cause is that function, the routing, serialisation, authentication middleware, the database, its migration state, the seeded data, the container's clock, a port collision, or the network. The same red bar carries ten times the investigation.

Why the difference is economic

The cost of a test is not its runtime. It is runtime plus the expected cost of diagnosing its failures, and the second term scales with the radius.

A 3-second test can run on every save, dozens of times an hour, so a defect is found while the change is still in your head. A 40-minute suite runs after the change has been merged and context-switched away from, so the same defect costs a context reload and a partial revert. The feedback loop, not the coverage percentage, is what the pyramid shape is about. Etsy's move to continuous deployment, documented from 2011 onwards with many deploys a day, only works because the signal that gates a deploy is narrow enough to act on in minutes.

What to do with it

  • Push a test down when its failures are usually caused by the narrow thing. If a broad test has failed eleven times this quarter and ten were a validation rule, that rule deserves its own fast test and the broad one can stop covering it.
  • Keep a test up when the risk is the wiring itself. Serialisation, auth middleware, SQL dialect, timezone handling and migration order cannot fail in a unit test, because a fake reproduces your beliefs about the dependency rather than its behaviour.
  • Judge by causes eliminated per second of runtime, which is why one broad smoke test per critical journey is worth a great deal and the four hundredth end-to-end test is worth very little.

When not to move a test down

Do not replace an integration test with a mocked unit test to make a suite faster if the bug class you are protecting against lives in the integration. The mock will agree with you forever. The honest alternative is to keep a small number of broad tests and delete the rest, rather than to keep the count and lower the fidelity: that trade costs you real defects to buy a green dashboard.