Leaderboard & Counting Service · View 16 of 21 · Operations
Decisions
- One write region. Multi-master counting would require either cross-region coordination on the hot path or a merge rule for conflicting deltas, and the product does not need either to tolerate a 10-minute RTO.
- The standby serves reads from an asynchronously replicated Bigtable at declared staleness, and rebuilds its cache on cutover. That is the whole reason RTO is 10 minutes rather than seconds, and it is an accepted cost.
- Spanner is multi-region because the things it holds — definitions, seasons, closed standings, audit — are the only state with RPO 0.
Numbers (assumptions)
- Ingestion accept ≥ 99.99% monthly; read path ≥ 99.95% monthly.
- RTO 10 minutes for the read path in the standby region; RPO 5 s for accepted events.
- 4× burst for 120 s absorbed by the log, with typed backpressure beyond it.
Risks
- A single write region means a regional failure stops counting, not just ranking. Clients must buffer and retry with the same idempotency keys, which is a contract on the product teams.
- Cold Memorystore on cutover means the first minutes after a failover run on the ranked store's latency, not the cache's.
