Distributed Job Scheduler  ·  View 05 of 20  ·  People and journeys

Journey — The Morning After an Outage

The on-call engineer who finds out from a customer, and what the platform owes them.

Editable source SVG draw.io All views
Tenant on-call paged at 07:40 Goal — Know what ran, what did not, and what is about to Trigger — A customer asks why last night's statement never arrived Done when — An answer good enough to tell the customer, without reading platform logs 1 · Notice 2 · Look 3 · Understand ◆ moment of truth 4 · Decide ◆ moment of truth 5 · Recover 6 · Prevent What they do Gets the complaint Opens the trigger Reads the skip cause Backfill, or let it go Requests a backfill Changes the policy What the platform shows Every instant, every state "expired: past horizon" Cost of the backfill Own lane, own rate cap Dry-run of the new policy How it feels Calm Tense Alarmed Where it hurts Found out from a customer Horizon was a default Backfill needs approval What answers it Lateness alert to tenant Scheduled vs dispatched Cause on every skip Privilege is the guardrail Original instant preserved Policy is declarable Journey — The Morning After an Outage The trough is "expired: past horizon" — a correct decision the tenant never agreed to. Everything in acts 3 to 7 exists to make that sentence explainable rather than surprising. v 1.0 · owner Platform Architecture · date 2026-10

What the trough says

  • The low point is the sentence "expired: past horizon" — a correct platform decision the tenant never consciously agreed to, taken on a default they did not read.
  • Everything in acts 3 to 7 exists to make that sentence explainable: a cause on every non-dispatch, the original scheduled instant preserved, and a backfill path that is privileged rather than impossible.

Decisions it forces

  • Fire history must distinguish scheduled, decided and dispatched instants, or "did it run" is unanswerable (ADR-13).
  • Backfill is a separate grant from authorship, because the recovery action is the one that multiplies real work (ADR-15).
  • A tenant-facing lateness alert, so the first notification is not a customer complaint.

Risks

  • The platform's own oldest-undispatched-due-age alert will be loud for hours while a legitimate backlog drains — the cost of making the backlog visible rather than hidden (Question 5 in the ask).
◀  Journey — Ship a Scheduled Job All views Layered Architecture  ▶
Solution Architecture v1.0 · Google Cloud · Platform Architecture · 2026-10 · 20 views · 16 architecture decision records05-journey-the-morning-after