Webhook Delivery Service  ·  View 05 of 20  ·  People and journeys

Journey — Endpoint Outage

Their service goes down at 02:00. Everything the platform does next is on this page.

Editable source SVG draw.io All views
Customer On-Call owns the receiving service Goal — Lose nothing while my service is down, and catch up safely Trigger — Their own 02:00 deploy takes the receiver offline Done when — Every event delivered once, nothing written to their ledger twice 1 · Outage begins 2 · Retries accrue 3 · Notified ◆ moment of truth 4 · Recover 5 · Catch up ◆ moment of truth What they do Deploys a bad build Asleep Reads the alert Rolls back Watches the backlog How it feels Calm Fine Alarmed Where it hurts No idea events queued Woken at 02:00 Replay could re-flood What the platform does Classifies as retryable Backoff, backlog held Circuit opens at 20 Alert on first dead letter Single probe succeeds Ramp 10% → 100% Same idempotency key Journey — Customer On-Call, Endpoint Outage The platform gives up loudly at the first dead letter, not at the hundredth. v 1.0 · owner Integration Platform Architecture · date 2026-09

What the platform owes them

  • Hold the backlog, and classify correctly: a connection refused is retryable, and retrying it for 72 hours is the product.
  • Stop hammering. The circuit opens at 20 consecutive failures and probes once every five minutes, so the platform is not adding load to someone else's incident.
  • Tell them at the first dead letter, through a channel independent of the broken endpoint — that is the actionable moment (ADR-09).
  • Ramp on recovery. An endpoint that just came back is the one least able to absorb a burst, so resumption goes 10% to 100% over five minutes (ADR-08).

The second moment

  • Catch-up is marked as a moment because a replay is a load test the customer runs against themselves. Rate caps, bulk confirmation, and the same idempotency key across every replay are what make it survivable (ADR-10).
  • The platform prefers a duplicate to a loss in every ambiguous case, and says so in the consumer contract rather than implying an exactly-once it cannot deliver (ADR-04).

Open

  • Auto-disable after 72 hours is the platform unilaterally breaking a customer's integration during that customer's own outage. It is defensible only because the backlog survives and the notification is reliable — and both of those are assumptions until measured (ADR-14).
◀  Journey — First Webhook All views Layered Architecture  ▶
Solution Architecture v1.0 · Amazon Web Services · Integration Platform Architecture · 2026-09 · 20 views · 16 architecture decision records05-journey-endpoint-outage