Event-Driven Notification Platform · View 26 of 26 · 6 · Assurance

Failure Modes and Disaster Recovery

Everything that can break, what covers it, and the one trade-off deliberately accepted.

Editable source SVG draw.io All views

The stated contract

At-least-once processing with idempotent consumers, and deduplication that suppresses duplicates where it can
The platform prefers duplicate processing over lost notifications, which is requirement nine restated as a design rule
No acknowledged event is lost: acknowledgement happens only after a durable quorum write

The accepted trade-off

If Redis is lost, deduplication fails open — the platform sends duplicates rather than dropping notifications
Flink keyed state and the unique dedup_key constraint still catch most of it, so the exposure is the window between them
Failing closed would drop transactional notifications during a cache incident, which is the worse outcome

Verification

Every mode on this page is exercised in a quarterly game day, not assumed
Broker kill, zone drain, provider blackhole and poison message are automated in the staging chaos run on every release train
Region failover is rehearsed twice a year with a real DNS cutover

◀ Identity and Access — Producer Authentication All views