beginner 2 min answer Multiple choice

One integration test fails only in CI, only on Tuesdays, only in the 00:30 UTC scheduled run. It passes locally and on every pull-request run. What do you look at first?

flaky-testsclockdiagnosisquarantineci
Pick one
Show the full answer Hide the answer

What the pattern tells you

A failure that correlates with a calendar has a calendar cause. Two facts narrow it immediately: it is time-locked (00:30 UTC) and day-locked (Tuesday). That is the signature of date arithmetic that assumes something about the current day — "the next business day", "start of week", "seven days ago", a fixture seeded relative to now(), or a window that behaves differently when the local date in the runner's timezone is the previous day. At 00:30 UTC, a runner or a service in a US timezone is still on Monday, so any code that mixes "today" from two clocks flips exactly here.

The order I would look in

  1. The failure's own timestamps and the values in the assertion message. A date off by one day or one hour ends the investigation in 2 minutes, which is why it costs nothing to look here first.
  2. Grep the test and its fixtures for the current time. now(), today(), LocalDate.now(), new Date(). Any of them in a test is a dependency on when it runs.
  3. Reproduce by pinning the clock, not by waiting a week. Run the test with the system time set to the failing moment, or with an injected clock. If it cannot be reproduced that way, the hypothesis was wrong and you have lost ten minutes rather than a sprint.
  4. Only then compare what is different about the scheduled run: the suite it executes, its data, its environment.

Why the other options fail

  • Stale cached dependencies. A real cause of scheduled-run-only failures, and it does not explain Tuesday. A cache is stale or it is not; it has no weekday.
  • A parallel-test race. The strongest distractor, because races are the most common flake class. It predicts random failures across runs, not a failure locked to one weekday and one clock time. Check it second if the clock hypothesis dies.
  • Staging timezone configuration. Plausible, and it usually produces a constant offset rather than a weekly pattern. It becomes the answer only when combined with a weekday boundary, which is the first option again.

The general rule this teaches

Flakes fall into five families: clock and calendar · test order and shared state · concurrency and races · resource limits and timeouts · external dependencies. The failure's pattern selects the family before any code is read, and time-locked failures are the cheapest family to confirm.

When quarantine is the wrong answer

Quarantine with no owner and no deadline is deletion with extra steps, and a quarantine list that grows every sprint is a suite people have stopped trusting. The same clock bug usually exists in production, where it fires once a week and nobody attributes the support ticket to it, so the test that looks like a nuisance is often the only thing reporting a real defect. Quarantine only with a named owner, an expiry date, and a rule that the test is fixed or deleted when it expires. A deleted test is honest about coverage; a permanently quarantined one lies about it.