On the last Sunday in October a UK data platform's 01:30 daily job ran twice and duplicated rows in a revenue table. On the last Sunday in March the same job did not run at all. The scheduler is configured in local time. What failed, and what is the structural fix?
Show the full answer Hide the answer
The trigger
On the last Sunday in October, UK clocks go back at 02:00 to 01:00. The hour from 01:00 to 02:00 occurs twice, so every wall-clock schedule inside it fires twice — including the 01:30 load. On the last Sunday in March the clocks jump from 01:00 to 02:00, that same hour does not exist, and schedules inside it never fire at all.
The exact window differs by zone: Central European Time transitions an hour later in local terms, so the exposed hour there is 02:00 to 03:00. The transition is not the bug. Scheduling against a clock that is not monotonic is, and teams that respond by shifting the job find that the next zone or the next rule change moves the window again.
Why it propagated
The duplicate run should have been harmless and was not. The load was an append keyed on a generated surrogate rather than a merge on the business key plus the logical date, so the second execution inserted a second copy of every row instead of replacing the first. Revenue for that date was double, and nothing errored.
Why detection lagged
Row-count checks compared the day against the previous day. The previous day was a Saturday, so the variance threshold was already wide, and a doubling landed inside it. The failure surfaced when Finance queried the month.
The structural fix
- Run the scheduler in UTC. Express business-day boundaries as a transformation applied inside the job — "the trading day is 00:00 to 24:00 Europe/London, resolved at query time" — rather than as the trigger's timezone. UTC has no repeated or missing hours.
- Make each task a pure function of its logical date, so a repeat is a no-op. The load becomes a merge on
(business_key, logical_date). A second execution finds the rows already present and changes nothing. - Add the constraint that makes a mistake loud. A uniqueness constraint or a post-load assertion on
(business_key, logical_date)turns a silent doubling into a failed run. This is the cheapest control here and the one most often missing.
The tempting local fix, and why it is not enough
Moving the job to 03:30 removes this year's exposure and none of the class. It survives until someone adds a market whose transition dates differ from Europe's — the United States changes on different Sundays, and much of the world does not change at all — or until a jurisdiction amends its rule, which happens somewhere most years. A platform with 300 local-time schedules has 600 exposures a year and no list of which ones matter.
The general lesson
A schedule is a business statement; a trigger is a machine statement. Keep the trigger on a monotonic clock and do the business conversion in the job, where it is visible, testable and can differ per market. The same discipline covers the adjacent trap: offsets that are not whole hours, such as India at UTC+05:30 and Nepal at +05:45, which quietly break any partitioning scheme that assumes "local midnight" and "a date boundary" are the same instant everywhere.
When this is not worth fixing
A platform in a single zone that does not observe daylight saving — much of India, China, Japan and most of Africa — has no exposure here at all, and converting 300 schedules to UTC to guard against a transition that does not occur is work with no return. The cost of the migration is not the configuration change; it is re-testing every partition boundary and every report that was written assuming local dates, which for a mature platform is weeks.
Prefer fixing the idempotency first in every case. It is one merge key per load, it costs about a day, and it converts the whole class of duplicate-execution faults — daylight saving, a scheduler restart, a manual re-run, a retried task whose worker was only presumed dead — from silent data corruption into a no-op. The timezone change removes one cause; idempotency removes the damage from all of them. That asymmetry is why the order matters, and it has been relearned by every platform that moved from a single-market schedule to a multi-market one since at least 2005.