Distributed Job Scheduler  ·  View 16 of 20  ·  Operations

Observability

Signal type by fire-path stage, and the one row that exists because the others cannot see this failure.

Editable source SVG draw.io All views
Declare Decide Commit Dispatch Outcome The alert Validation failure rate Oldest undispatched due age > 60 s pages Ledger write errors Lateness p99 breach Unknown-state fraction Metrics Definitions / s Propagation p99 Clock ε per node Leases vs partitions Fires committed / s Duplicate-key rate Lateness histogram Queue depth per lane Success by target class Traces Validate span Scan → claim span Commit span Attempt span per n Callback correlation Tenant-facing Next 3 instants Skip cause Fire record Per-trigger lateness Terminal state Audit Definition change Lease transfer Manual run Backfill grant Quota change Reporting Trigger population Shadow divergence Fire counts for billing Peak avoided by jitter Cost per million fires Observability — Signal Type by Fire-Path Stage Oldest undispatched due age is the only signal that catches the characteristic failure: a scheduler that is up, healthy on every other dashboard, and not firing. v 1.0 · owner Platform Architecture · date 2026-10

Decisions

  • The page is on oldest undispatched due age, not on process health. A scheduler that is up and not firing is healthy on every other dashboard — this is the characteristic failure of the system (ADR-13).
  • Clock ε is a per-node metric, because a fleet-wide drift is invisible in an average and is one of the two findings that would change the design.
  • Tenant-facing signals are a product surface with their own row, so "did it run" never requires platform access.

Assumptions

  • Oldest undispatched due instant older than 60 s in any partition pages immediately. Assumed, and knowingly noisy during a legitimate catch-up drain.

Risks

  • Shadow divergence is reported, not alerted on, in the MVP. A divergence that appears between reports is a window in which the rebuild claim is unverified.
◀  Deployment Architecture All views Trigger Lifecycle  ▶
Solution Architecture v1.0 · Google Cloud · Platform Architecture · 2026-10 · 20 views · 16 architecture decision records16-observability