fail_mode is a column on the organisation, because the right answer depends on what the tenant is paying for. An enterprise tenant on a revenue-critical path wants FAIL OPEN; a free tenant that could run up a provider bill wants FAIL CLOSED. A single global constant is wrong for one of them.
The default is LOCAL, not open or closed. Each pod falls back to quota ÷ pod count × 0.8, which keeps enforcement approximately correct instead of abandoning it — the right answer for the large middle of the tenant base.
PostgreSQL is not on the hot path, so its outage degrades only the ability to change policy. That is the clearest payoff of the control-plane split in view 03, and it is why a policy-store failure is an inconvenience rather than an outage.
Blast radius, quantified
Valkey shard loss: overshoot up to 15% for tenants on that shard during LOCAL fallback, bounded to one window; other shards unaffected.
Policy bus lag: pods serve last-known-good policy indefinitely. Correct behaviour, stale limits, alerted above 60 s.
Provider outage: failover within 5 s for tenants that permit it, 503 with retry_after for tenants pinned by policy.
Complete Valkey cluster loss: HIGH tenants continue at full local quota, STANDARD at 80%, FREE receive 429.
Recovery
Counters are not repaired after a shard returns; they carry a two-window TTL and rebuild correct state within 500 ms. Repair jobs for rate-limit counters are a source of bugs and are deliberately absent.
The reaper sweeps holds leaked during the incident on its normal 1 Hz cycle; no manual step.
The usage ledger is rebuilt by replaying Kafka within the 7-day retention window. Billing is eventually correct even when live enforcement was degraded.
Each mode is exercised in a quarterly game day, including a full cluster loss, because a fallback path that is never run is a fallback path that does not work.
