Leaderboard & Counting Service  ·  View 17 of 21  ·  Operations

Observability Matrix

Six signal classes across five stages, and the one signal a member can see.

Editable source SVG draw.io All views
Admit Log Aggregate Project Serve Latency Accept p99 target 25 ms Publish confirm Window close delay Build duration Top-N p99, rank p99 Freshness Log backlog seconds Pipeline watermark lag Projection lag per board the user-visible one As-of age served Correctness Dedup hit rate Poison sink volume Late-event ratio Rebuild diff vs live Rank monotonicity breaks Skew & capacity Top-key write share Partition imbalance Shard count per key Scopes materialised Cache hit ratio Abuse & integrity Quarantine volume Lockstep clusters Frozen scopes Retractions served Cost $ per M events Archive GB by class Worker hours Rebuild cost per run $ per M reads Leaderboard & Counting Service — Observability Matrix Projection lag has its own alert and its own owner, because it is the only signal here that a member can see. Everything else in this matrix is a cause. v 1.0 · owner Platform Architecture · date 2026-10

Decisions

  • Projection lag per leaderboard has its own alert and its own owner, because it is the only row in this matrix a member can perceive. Everything else here is a cause.
  • Rebuild diff against live is a routine correctness signal, not an incident artefact: a scheduled rebuild that disagrees with the live projection is the earliest possible warning that an accumulator is not associative.
  • Rank monotonicity breaks are tracked as a correctness signal, because a rank that oscillates within a session is the failure members report and no latency metric shows.

What the matrix is for

  • Every column is a stage that can be the cause; every row is a way the platform can be wrong. A cell that stays empty is a statement that the stage cannot fail that way.
  • Cost sits in the matrix rather than in a monthly report, because the exact-versus-approximate decision is only honest if its price is visible continuously.

Risks

  • Six signal classes × five stages is a lot of alerting surface for a 9-person team. Only projection lag, quarantine volume and the infrastructure error rate should page.
  • Top-key write share is a leading indicator that only helps if the adaptive splitter acts on it faster than a launch can arrive.