# Distributed Job Scheduler

**Solution Architecture v1.0 · Google Cloud · Platform Architecture · 2026-10 · 20 views · 16 architecture decision records**

A developer opens the settings screen, types `0 9 * * 1-5`, picks a time zone, and expects a digest to go out every weekday morning. The box is four words wide. Behind it is a durable multi-tenant timer fleet that has to fire tens of millions of independent triggers at the right instant while a partition owner is being replaced, a node's clock is wrong, a zone is draining, and a tenant has just resumed four thousand triggers that slept through a day. This package is that service: the scheduled-trigger platform inside one developer product — an assumed 50,000 tenants, 20 million active triggers, 500 million fires a day at a mean of 5,800 a second, with 60% of all fires landing in the first second of a minute and a design peak of 45,000 a second for five seconds at the top of the hour — built on a globally consistent relational store for the registry, the due index and the fire ledger with commit timestamps as time authority, a rate-shaped task queue for lane-split dispatch, serverless containers for the control, timing and dispatch tiers, a wide-column store for fire history, and a columnar warehouse for lateness and cost reporting.

The product is not the ability to fire a timer. It is **the ability to be interrupted** — because a scheduler will lose a node, a clock, a zone and a day, and the only question that matters is whether what it owes afterwards is a declared policy or an improvisation.

The design rests on one rule: **the timing plane decides, the execution plane works, and the only thing crossing between them is an immutable, uniquely-keyed fire record that is committed before it is delivered.**

The decisions that carry the design:

- **Commit before dispatch, always.** The fire record is written before any attempt, so a dispatcher that dies mid-attempt loses an attempt and never a fire. Of the two orderings available, one failure is unrecoverable and the other is a retry — and the order of two writes decides which one the system can have (ADR-01).
- **The console cannot take the fire path down.** Two planes, separately deployed and scaled, sharing only the state of record. One write region, because two regions evaluating "is the previous run still going" produce two answers that no key reconciles (ADR-02).
- **Idempotency lives in the data model.** The fire's primary key is (tenant, trigger, scheduled instant, sequence), computed rather than generated. Split ownership becomes a uniqueness violation rather than a detection problem, which is exactly what lets leases be short, takeover fast, and leader election imperfect (ADR-03).
- **Late is a budget; early is a bug.** The node clock is untrusted. The scheduler takes a bounded uncertainty interval, waits it out, and scans for instants due at or before its lower bound — paying about 10 ms of deliberate lateness to make earliness impossible, because scheduled work reads windows of data that do not exist yet (ADR-04).
- **Recurrences are recomputed, not materialised.** One due-index row per trigger, holding one next instant. An amendment, a pause, a validity change and a time-zone database update are all picked up with no invalidation sweep, because nothing was cached to invalidate — and dry-run is exact rather than indicative (ADR-05).
- **The post-outage decision is declared before the outage.** Each trigger declares its missed-fire policy from a closed set of three, defaulting to firing once for the most recent missed instant. A billing run and a cache warm have opposite correct answers and look identical to the platform (ADR-06).
- **Wall-clock time means what the tenant meant.** Expression plus named zone, resolved at computation, with declared rules for the day a time does not exist and the day it happens twice, and the time-zone database version recorded on every fire so a disputed instant is explainable from data (ADR-07).
- **Every policy is bounded by a horizon.** An instant older than the catch-up horizon is recorded as expired with its cause and never dispatched, whatever the tenant declared — the one place the platform overrules them. Caught-up fires carry their original scheduled instant, because the work reads a window defined by it (ADR-08).
- **Recovery is slower than failure, by design.** Three queues with independent drain rates, a per-tenant catch-up cap, and a published shedding order. A one-hour backlog takes about ten hours to drain, which is the price of not making the recovery into the next outage for every executor downstream (ADR-09).
- **The peak is absorbed, not purchased.** Deterministic opt-out jitter smearing and the lateness budget absorb a peak that lasts five seconds an hour, instead of provisioning eight times the steady-state capacity to be idle for the rest of it (ADR-10).
- **Rebuildable means rebuilt on a schedule.** The due index, the leases and every cache are projections of the registry and are not backed up. A shadow rebuilder recomputes and compares continuously, because a recovery path exercised only in an incident is a claim rather than a capability (ADR-11).
- **`unknown` is a state, not an absence.** Overlap is enforced against recorded outcome state bounded by a window. Treating a missing callback as "still running" turns one lost message into a permanently stopped trigger that looks perfectly healthy (ADR-12).
- **Alert on the gap, not on health.** The paging signal is oldest undispatched due instant age, because a scheduler that is up and not firing passes every other check. The same measurement, per fire, is a tenant-facing surface — so "did my job run" is answerable without a support ticket (ADR-13).
- **A dispatch target is a privilege, not a parameter.** Targets must be verifiably owned, checked at definition and again at dispatch; no tenant request reaches the fire zone; credentials are minted per attempt. A service that makes authenticated requests to any URL on a timer with a retry budget is an attack tool until ownership is proved (ADR-14).
- **The operations that multiply work are separate grants.** Backfill, horizon extension and quota change are grantable apart from authorship and audited with prior values, because each is most attractive at the moment judgement is worst (ADR-15).
- **Opaque data gets the strictest boundary.** The tenant payload exists in two stores under a per-tenant key and never in history, reporting, metrics or logs. The platform cannot classify what it does not interpret (ADR-16).

The architecture one-pager (including why the design should still hold up in ten years, and the five risks that would change it) and the full decision record appear on the landing page of the diagram set, directly below the index of views. The same content is published as [docs/architecture-one-pager.md](docs/architecture-one-pager.md) (~18 min) and [docs/decision-record.md](docs/decision-record.md) (~63 min).

---

## What is here

| Path | Contents |
|---|---|
| `diagrams/index.html` | The landing page: 20 views in seven acts with every format linked, then the **architecture one-pager** and the **decision record** |
| `diagrams/*.html` | One self-contained page per view: the inlined diagram plus the reasoning cards, with copy / PNG / PDF export |
| `diagrams/svg/*.svg` | The same views as SVG with the diagram XML embedded; they re-open fully editable in diagrams.net |
| `diagrams/drawio/*.drawio` | draw.io source |
| `docs/architecture-one-pager.md` | The one-pager as markdown |
| `docs/decision-record.md` | The 16 decision records, the capability-to-technology table and the package glossary as markdown |
| `specs/part-a..c.json` | Diagram specifications, the source of truth for every view |
| `specs/manifest-a..b.json` | Acts, page titles, subtitles and reasoning cards |
| `specs/adr-onepager.json`, `specs/adr-records-a..b.json` | The one-pager, the decision records, the capability-to-technology table and the glossary |
| `scripts/build.sh` | Rebuilds every deliverable from the specs (Node only, no network) |
| `ask.md` | The requirement |

## The twenty views

| # | View | What it answers |
|---|---|---|
| 01 | System Context | Who declares schedules, where the work actually runs, and the much larger thing the service refuses to own |
| 02 | High-Level Architecture | Seven stages on one spine, with the ledger commit sitting between deciding and delivering |
| 03 | Actors and Their Core Journeys | Six parties, two of them machines, including the time authority whose requirement is to be believed rather than averaged |
| 04 | Journey — Ship a Scheduled Job | The trough is not the cron expression: it is the policy screen, the one moment a developer is asked a distributed-systems question |
| 05 | Journey — The Morning After an Outage | The trough is "expired: past horizon" — a correct decision the tenant never consciously agreed to |
| 06 | Layered Architecture | Seven layers, and the single seam: the control plane and the timing plane share only the state of record |
| 07 | Platform Components | The containers in one region, and the fact that exactly one of them has egress |
| 08 | Integration Surface | Three inbound APIs separated by their authorisation stories, and four outbound target classes |
| 09 | Data Flow — Definition to Evidence | Seven states of one instant, of which only two are authoritative |
| 10 | Storage Zones | Backed up, rebuildable, or cold — and why that is the only classification that matters here |
| 11 | Data Model | Eleven entities, and one four-part primary key doing the work of a deduplication service |
| 12 | Critical Flow — One Instant, Decided and Delivered | Eighteen messages, including the duplicate-key rejection that is a normal return rather than an error |
| 13 | Catch-up After an Outage | The expected failure mode drawn as a pipeline, with the horizon filter ahead of the policy |
| 14 | Dispatch Lanes | Five classes of fire through one pipeline, with a published order for who loses capacity first |
| 15 | Deployment Architecture | One write region across three zones, and a standby that deliberately holds no leases |
| 16 | Observability | Signal type by fire-path stage, and the one row that exists because the others cannot see this failure |
| 17 | Trigger Lifecycle | Seven states, and why the loop only closes if the tenant can see its own lateness |
| 18 | Security Trust Zones | Five zones, and the guard that stops a schedule becoming a timer-driven request-forgery primitive |
| 19 | Identity and Privilege Flow | Who proves what, in what order — including the horizon extension that is deliberately refused |
| 20 | Failure Classes and Their Answers | Ten classes with their structural answer and their accepted residual, plus the two that would change the design |

## Rebuilding

```bash
bash scripts/build.sh
```

Node 20+ and nothing else. No draw.io Desktop, no browser, no network. The build assembles
`specs/views.json` and `specs/manifest.json` from the authoring parts, generates the draw.io
sources, fails on any geometry or routing defect, renders the editable and plain SVGs, writes the
HTML pages and the index, injects the one-pager and the decision record, and proves every relative
link in `diagrams/` resolves.

At v1.0 the geometry gate reports 0 errors and 0 warnings across 20 files, the routing gate
0 errors and 6 clutter warnings across 169 edges, all 351 nodes resolve to an icon with no weak
matches, and the link check passes on 158 relative links. The 6 warnings are label stacking on the
integration, container and trust-zone views, where several relationship edges converge on one
centre; they were reduced from 26 by shortening labels and cutting edges that other views already
carry, and the remainder is inherent to those layouts.

## A note on the numbers

Every rate, latency, ratio, threshold and retention figure in this package is a **stated
assumption**, chosen to be defensible and arguable rather than measured. They are stated precisely
so that a reviewer can disagree with one and follow it to the decision that depends on it.
`ask.md` marks them as assumptions section by section; the decision record's evidence note
restates the operating context in one paragraph.