Webhook Delivery Service  ·  View 14 of 20  ·  Runtime

Replay and Recovery

Five lanes, and the reason bulk replay needs a confirmation dialog.

Editable source SVG draw.io All views
Detect Decide Authorise Re-deliver Confirm Customer on-call Reads failure alert Fixes the receiver Chooses the scope Watches it drain Console and API Delivery health API Dead-letter browser Confirm bulk replay Request inspector Replay service Counts the backlog Per-endpoint rate cap Same idempotency key Marks replayed_as Delivery plane Circuit open Single probe succeeds Ramp 10% → 100% Endpoint healthy Consumer's system Receiver down Receiver restored Dedups on the key Caught up Replay and Recovery — Who Does What A bulk replay is a load test the customer is running against themselves, which is why it is rate-capped and confirmed. v 1.0 · owner Integration Platform Architecture · date 2026-09

Decisions

  • A replay is a new delivery carrying the original event id and the original idempotency key, so the consumer's deduplication still protects them (ADR-10).
  • Rate-capped per endpoint, and a bulk replay above a declared size requires explicit confirmation. A four-day backlog replayed at full speed does to a customer's system exactly what the original outage did.
  • Replay of successfully delivered events within the retention window is supported. 'We processed it and then lost our database' is a real request and is not a failure of this platform.
  • Replay workers are a separate pool, so a bulk replay cannot starve first-attempt delivery.

The recovery ramp

  • Circuit probe succeeds, then resumption ramps from 10% to 100% of the endpoint's concurrency cap over 5 minutes (assumption).
  • This is the platform declining to be the cause of the customer's second outage, and it is the requirement most likely to be removed by someone optimising drain time.

Not drawn

  • The skip-the-backlog option on re-enable. It exists, and its consequences belong with the endpoint lifecycle in view 18.
  • Cross-region replay. Deferred to Phase 2 with the rest of multi-region delivery.