beginner 3 min answer Multiple choice

A team merges to trunk several times a day and the main branch's build is broken for two or three days at a time. They have a CI server, a large test suite and a merge queue. Which change restores continuous integration?

continuous-integrationtrunkrevertbuild-healthdiscipline
Pick one
Show the full answer Hide the answer

The mechanism

Continuous integration is a behaviour, not a server. The property that makes it valuable is that the trunk is always a candidate for release, and the property that destroys it is a broken trunk that stays broken, because every subsequent change is now built on an unknown base. Failures compound: after two days nobody can tell whether their test failure is theirs.

A revert-first rule attacks the duration directly. The commitment is that the trunk returns to green within a stated time — commonly 10 to 30 minutes — by reverting the offending change if it is not fixed by then. The change goes back to its author with a failing test, which is a normal day's work, instead of becoming a shared emergency.

Two things make it survivable. Reverting must be socially neutral, which requires saying out loud that a revert is not a judgement, or engineers will resist it and the rule dies. And changes must be small enough to revert, which is why this rule and small batches arrive together: a 40-file change touching a migration cannot be reverted in ten minutes, and the cure is upstream in how the work was split.

Why the other options fail

  • More pre-merge tests. Helpful against some breakages and it addresses the wrong variable. The problem in the question is the duration of red, not the rate of it, and a longer pipeline makes duration worse by slowing the fix's own verification. It also has a hard limit: integration failures and flaky infrastructure survive any pre-merge suite.
  • Work on branches until green. This is the natural instinct and it converts the problem into merge debt. Branch lifetimes grow, everyone integrates at once when the trunk recovers, and the resulting merge storm breaks the trunk again. It is the definition of not doing continuous integration, dressed as prudence.
  • A rotating build cop. A genuinely useful role, and it treats the symptom: someone is now responsible for a state that should not persist. Build cops work well alongside a revert rule and, on their own, they institutionalise a red trunk by making it somebody's job.

What to measure

Two numbers, published weekly: time to green after a break, at p95, and the share of the week the trunk was red. A team that cannot state these is not managing the thing that is broken. If time-to-green is hours, the revert rule is not being followed, and the reason is almost always that changes are too large to revert.

When this is the wrong answer

If the failures are flaky rather than real, reverting is noise — you will revert innocent changes and the trunk will still be red. Flakiness is a separate problem with a separate fix: quarantine the identified flaky test with an owner and a deadline, track the flake rate, and never let a flaky suite be the reason the team stops trusting failures. The order matters: make failures meaningful, then make them short-lived. And a team of three sharing one repository can often keep the trunk green by shouting across a room, which is a perfectly good mechanism at that size.