Leaderboard & Counting Service  ·  View 21 of 21  ·  Assurance

Assumed Failure Classes

Seventeen named failures in three tiers, sorted by who notices.

Editable source SVG draw.io All views
Absorbed in the write path — the member never learns Delivery Duplicate delivery idempotency key Out-of-order events event-time windows Late beyond horizon visible late bucket Shape of the load Hot key, 1000× median adaptive sharding 4× burst, 120 s log admits, typed shed Clock skew server ingest time kept Visible as staleness, never as a wrong number Serving Cache tier lost fall through, 150 ms Aggregation stalled as-of ages, flagged Region unavailable standby reads, RTO 10 m Projections Bad ranking deploy serve previous version Partition write loss replay that partition Poison event sink, pipeline continues Requires a human decision, and leaves a record Integrity Inflation campaign quarantine + freeze Disqualified member bulk compensation Retraction after closure correction record Configuration Incompatible definition refused, new version Tenant over quota shed, not spill Not handled: a product emitting wrong events the platform counts what it is told Leaderboard & Counting Service — Assumed Failure Classes Application we own Risk / gap Security / platform Decision point Three tiers by who notices: nobody, the reader as a widened as-of, and an analyst who must decide. The platform's job is to keep as much as possible in the first tier and nothing silently in the third. v 1.0 · owner Platform Architecture · date 2026-10

How the tiers are drawn

  • Tier 1 is absorbed in the write path and nobody learns: duplicates, out-of-order events, hot keys, bursts, clock skew.
  • Tier 2 is visible only as staleness, never as a wrong number: a lost cache, a stalled pipeline, a lost region, a bad projection build, a poison event.
  • Tier 3 requires a human decision and leaves a record: an inflation campaign, a disqualification, a retraction after closure, an incompatible definition.

The design rule

  • Keep as much as possible in tier 1, and nothing silently in tier 3. Every tier-3 entry has an actor, an authority and a record.
  • "Flake" is not in this catalogue. A failure with no named class is a failure nobody owns.

Outside the platform's reach

  • A product that emits wrong events. The platform counts what it is told, and an event the product should never have sent is counted correctly and wrongly at the same time.
  • A tenant whose own definition of a point changes mid-season. Versioning the counter records it; it cannot make the season comparable.
◀  Who May Increase a Counter All views
Solution Architecture v1.0 · Google Cloud · Platform Architecture · 2026-1021-failure-classes