beginner 3 min answer Multiple choice

A test passes when run on its own and fails when the whole 900-test suite runs. The error says a user with that email already exists. CI runs tests in file-discovery order so the failure is perfectly repeatable. What is the most useful first move?

test isolationorder dependenceshared statebeginnerbisection
Pick one
Show the full answer Hide the answer

The diagnosis

The failing test is the victim, not the culprit. An earlier test created a user with that email and did not remove it. The test under investigation is correct and is reporting someone else's mess, which is why reading its code teaches you nothing.

Two properties of the situation are doing all the work. Shared mutable state - one database, rows that outlive the test that made them - means tests can affect each other. Stable discovery order means the interaction is reproducible but invisible: it looks like a property of the failing test rather than of a pair.

Why the proposed move is the first one

A recorded random seed makes order-dependence a first-class, reproducible fact. Run the suite with a seed, get the failure, then bisect the ordering: keep the victim last, halve the set of tests that run before it, repeat. With 900 tests there are too many ordered pairs to search by hand, but bisection finds the polluter in about ten runs because each run halves the candidate set. The major frameworks support this directly, with a seed printed on every run so a failure can be replayed exactly.

The fix that follows is almost always the same shape: the polluting test creates its data through the product's write path and removes it, or the suite wraps each test in a transaction that is rolled back. Both make the suite order-independent, which is the property you actually want - order dependence is a defect in the suite, and a suite that has one almost certainly has several.

Why the other options fail

  • Quarantine it. The right response to a genuinely non-deterministic test, applied here to a deterministic one. The suite goes green and the pollution stays, so the next test that touches that email fails too. You have hidden a defect in test isolation that will keep producing victims, and the quarantine list grows until nobody believes it.
  • Defensive cleanup at the start of the test. It works, which is the trap. The victim passes, the polluter still leaks, and the codebase accumulates a defensive preamble in every test that shares the fixture. Three years later no test can be read without knowing what the others leave behind.
  • Drop to one worker. Reasonable if the symptom were a deadlock, a port conflict or a flaky parallel setup - and it would be the right call then. Here the failure is sequential ordering, not concurrency, so serialising changes nothing except an eightfold increase in wall-clock time, which is a real cost paid for no information.

When this is the wrong answer

When the suite is already randomised and the failure tracks the worker count rather than the order. Then the shared state is across processes - a fixed port, one schema used by eight workers, a shared cache - and order bisection will chase nothing for a day. The test to tell them apart is cheap: run the suite twice with the same seed and a different worker count. If the failure follows the seed, bisect the order. If it follows the worker count, give each worker its own schema.