Webhook Delivery Service  ·  View 12 of 20  ·  Runtime

Retry, Circuit Breaking and Dead Letter

Four outcome classes, one published schedule, and two ways back in.

Editable source SVG draw.io All views
Attempt Delivery Worker 5 s / 10 s / 15 s Classify outcome Success 2xx Retryable 5xx, 408, 429, timeout Permanent other 4xx Rejected address check, disabled Schedule Backoff Scheduler full jitter, 5 s → 6 h Retry-After honoured to 1 h Circuit Breaker opens at 20 failures Exhaust 12 attempts or 72 h whichever first Auto-disable 72 h, zero success Account and recover Dead-Letter Store 30 d, backlog held Owner Notification on the first dead letter Replay single, filtered or bulk next attempt re-deliver Retry, Circuit Breaking and Dead Letter Application we own Opportunity Decision point Risk / gap Data store Interface / broker event / async A permanent failure is never retried, and never silent: its owner is told, because a quietly dead integration is the worst outcome here. v 1.0 · owner Integration Platform Architecture · date 2026-09

Decisions

  • Four outcome classes, published: success, retryable, permanent, rejected-before-dispatch. A developer's first question is which class their failure landed in, and an unclassifiable failure is a support ticket.
  • The retry schedule is the platform's, not the customer's: 12 attempts over 72 hours, full-jittered exponential from 5 s, capped at 6 h. Customer-configurable retries let a tenant configure a self-inflicted denial of service and make capacity unplannable (ADR-08).
  • `Retry-After` is honoured up to an hour, because a consumer telling the platform when to come back is the most useful signal it will ever get.
  • A permanent failure is never retried and never silent. Its owner is told, because a quietly dead integration is the worst outcome this platform can produce (ADR-09).

Numbers, all assumptions

  • Circuit opens at 20 consecutive failures; probes once every 5 minutes with a single attempt.
  • Auto-disable after 72 hours of continuous failure with zero successes; the backlog is retained for the dead-letter window regardless.
  • Retries assumed at 12% of the 45,000 attempts/s steady state. Above 25% an alarm fires, because retry traffic is capacity spent producing nothing.

Open

  • Auto-disable is the platform breaking a working configuration during the customer's own incident. It is justified only if the notification is reliable and the backlog survives — see ADR-14 for the argument and what would change it.
  • Whether named retry profiles should replace the single fixed schedule is deferred to Phase 2 rather than settled here.
◀  Critical Flow — Event to First Attempt All views Signing, Rotation and Verification  ▶
Solution Architecture v1.0 · Amazon Web Services · Integration Platform Architecture · 2026-09 · 20 views · 16 architecture decision records12-retry-and-dead-letter