A service suite of 1200 tests takes 9 minutes. CPU on the CI runner sits at about 20% the whole time. A profile shows each test spends roughly 350 ms creating a schema and seeding reference data before it runs; the assertions themselves take under 5 ms. Which change do you make first?
Show the full answer Hide the answer
The arithmetic first
1200 tests at 350 ms of setup is 420 seconds - 7 of the 9 minutes, about 78% of the run. The assertions account for 6 seconds in total. Whatever you do to the tests themselves cannot win more than 2 minutes; the setup is the suite.
The 20% CPU figure is the confirming clue: the runner is waiting, not computing. Schema creation is DDL plus catalogue writes plus seed inserts, and the test process is idle for most of each 350 ms.
The diagnosis
Per-test schema creation is the default in a lot of test harnesses because it is the simplest way to get isolation, and isolation is the property you must keep - tests that see each other's rows fail in whichever order exposes the dependency. The question is how cheaply you can buy it.
A transaction gives it almost free. Migrate once per run, then begin a transaction before each test and roll it back afterwards. The rollback discards the test's writes in microseconds rather than hundreds of milliseconds, because nothing was ever made durable. Expect the suite to land near 1 to 2 minutes.
Where the harness cannot wrap a transaction - a test that spans processes, or code that manages its own transactions - the next cheapest option is a template database: build the schema once and create each test database from that template, which copies files instead of replaying DDL and seeds.
Why the other options fail
- Eight parallel workers. Real, and it treats the symptom. Wall clock falls, total compute does not, so you pay roughly eight times the runner minutes for the same work - and you still carry 7 minutes of setup per full run. Parallelism also needs isolation you have not built yet: the first thing it exposes is tests that share state, which arrive as flakes nobody can reproduce locally.
- An in-memory substitute for the database. The most expensive mistake on the list, because it looks like the biggest win. A different engine does not enforce the same constraints, does not have the same SQL dialect, and does not run your migrations. Every defect that lives in a unique index, a cascade, a partial index or a window function becomes invisible to the suite and ships.
- One shared seeded database. Fast and it destroys the property the setup was buying. Tests become order-dependent, a deleted test breaks an unrelated one, and the suite can no longer be run as a subset - which is the thing you want on the day you are debugging.
What to keep after the change
- A handful of tests that really commit - a few per cent of the suite - because commit-time behaviour (deferred constraints, triggers, after-commit hooks, isolation anomalies) is exactly what a rolled-back transaction hides. Mark them and let them pay the full reset.
- Per-test data built by factories, creating only what the test needs. Large shared seed data is the other half of the 350 ms and it also makes tests depend on rows they never mention.
- The suite time as a tracked number. Feedback latency sets how often engineers run the suite at all, so it belongs on a dashboard rather than in anecdotes.
When this is the wrong answer
At small scale, do nothing. Thirty tests at 350 ms is 10 seconds of setup; the fix costs more than it returns. The threshold is the point where setup time dominates the run and engineers have started avoiding the suite - which in practice is around the two-minute mark, not a test count.
And if the suite is slow because it starts six containers for every test, the setup is not the schema and the answer is different: decide what those tests are for, and push most of them below the process boundary.