Event-Driven Notification Platform · View 17 of 26 · 4 · Runtime
Retry, Failover and Dead-Lettering
What happens when a provider says no, and how a message that cannot be sent stays recoverable.
Copy
PNG
PDF
⋯
Editable source
SVG
draw.io
All views
Email Worker
Email Worker
Rate Limiter
Rate Limiter
Postal MTA
Postal MTA
Amazon SES
Amazon SES
Delivery Store
Delivery Store
Kafka DLQ
Kafka DLQ
Alertmanager
Alertmanager
1. acquire token
1. acquire token
2. granted
2. granted
3. submit · attempt 1
3. submit · attempt 1
4. 451 transient
4. 451 transient
5. attempt 1 · TRANSIENT
5. attempt 1 · TRANSIENT
6. backoff 2^n plus jitter
6. backoff 2^n plus jitter
7. submit · attempt 2
7. submit · attempt 2
8. timeout
8. timeout
9. circuit opens · 5 fails in 30 s
9. circuit opens · 5 fails in 30 s
10. failover submit · attempt 3
10. failover submit · attempt 3
11. 250 accepted
11. 250 accepted
12. attempt 3 · SENT via secondary
12. attempt 3 · SENT via secondary
13. half-open probe after 60 s
13. half-open probe after 60 s
14. 550 permanent · mailbox unknown
14. 550 permanent · mailbox unknown
15. park with full context
15. park with full context
16. DLQ depth breach
16. DLQ depth breach
Delivery Failure — Retry, Failover and Dead-Lettering
Delivery Failure — Retry, Failover and Dead-Lettering
Transient failures retry then fail over; permanent failures go straight to the DLQ without spending the retry budget. Five attempts over 30 minutes for P1, two over 60 seconds for P0.
Transient failures retry then fail over; permanent failures go straight to the DLQ without spending the retry budget. Five attempts over 30 minutes for P1, two over 60 seconds for P0.
v 1.0 · owner Data & AI Global Practice · date 2026-08
v 1.0 · owner Data & AI Global Practice · date 2026-08
Text is not SVG - cannot display
Failure classification
Transient — timeouts, 5xx, 4xx throttling — retried with exponential backoff and jitter
Permanent — invalid address, unsubscribed, hard bounce — straight to the DLQ, no retry budget spent
Misclassifying a permanent failure as transient is the most expensive bug in this design, so the mapping is per provider and unit-tested
Budgets
P0: two attempts within 60 seconds, then failover, then DLQ — speed matters more than persistence
P1: five attempts over 30 minutes across both vendors
P2: three attempts over 4 hours, and shed entirely under backpressure
Circuit opens after 5 failures in 30 seconds, half-open probe after 60 seconds
Assumptions
Both vendors for a channel are contracted and warm; a cold failover vendor makes these numbers fiction
DLQ messages carry the full rendered payload, which means the DLQ holds PII and is encrypted and access-controlled accordingly
◀ Timing, Scheduling and Digests
All views
Deployment and Failure Domains ▶
