Distributed Job Scheduler  ·  View 20 of 20  ·  Assurance

Failure Classes and Their Answers

Ten classes, each with what it looks like, what answers it structurally, and what is left over.

Editable source SVG draw.io All views
Assumed to fail How it shows Structural answer Residual risk Clock Skew, step, lost sync ε above bound Shed leases, wait out ε Fleet-wide ε drift Owner loss Death between claim and send Lease expiry Re-derive from registry One duplicate attempt Split ownership Two owners mid-handover Duplicate-key rate Fire key uniqueness Executor must dedupe State unavailable Registry or index down Rising due age Fail closed on dispatch Stall, then catch-up Target Transient, 4xx, systematic Success by target class Budget, stop, isolate Tenant unaware of 4xx Poison definition No future instant Error on the trigger Quarantine one trigger Slow compiler path Catch-up storm Backlog released at once Backlog depth per tenant Horizon, cap, own lane Alert noisy while draining Slow executor Run outlives the interval Unknown-state fraction Overlap policy + window Concurrent run, or silence Zone or region Zone loss, region loss Degraded lateness p99 Reassign; declared promotion 15 s of decisions at RPO tzdata change Future instant moves Version on each fire Recomputed instant governs Surprise at the boundary Assurance — Failure Classes and Their Answers Every row's residual risk is accepted knowingly. The two that would change the design are fleet-wide clock drift and an executor that cannot deduplicate. v 1.0 · owner Platform Architecture · date 2026-10

How to read it

  • Every row's residual risk is accepted knowingly. A resilience section without a residual column is a list of patterns pretending to be a design constraint.
  • Three answers do most of the work: the fire key (split ownership, owner loss), the clock gate (clock failure), and the lane split with its rate cap (catch-up storm).
  • Fail-closed-on-dispatch is the deliberate choice for state unavailability: the service stalls visibly rather than firing on a guess.

The two that would change the design

  • Fleet-wide clock drift — if ε cannot be bounded per node, the lease-and-gate arrangement in views 06 and 12 has no foundation and the design needs a different authority.
  • An executor that cannot deduplicate — at-least-once stops being a contract and becomes a defect, which forces either exactly-once machinery or a platform-side suppression store.

Assumptions

  • Each class is assumed to occur independently. Correlated failures — a zone loss during a catch-up drain — are the gap this view does not cover and a game-day exercise should.
◀  Identity and Privilege Flow All views
Solution Architecture v1.0 · Google Cloud · Platform Architecture · 2026-10 · 20 views · 16 architecture decision records20-failure-classes