# Architecture One-Pager

*Webhook Delivery Service · Solution Architecture v1.0 · Amazon Web Services · Integration Platform Architecture · 2026-09 · 20 views · 16 architecture decision records*

**Capture and delivery are separate planes joined by a durable log, and the unit of isolation on the delivery side is the endpoint — not the tenant, not the event type, and not a shared worker pool.**

A SaaS product with an active ecosystem has to call 40,000 servers it does not own, cannot page, and must not trust. Some of them are behind a laptop on a hotel connection. Some return 200 OK and then throw the event away. One of them, at any given moment, is taking fifteen seconds to respond and never erroring — and that one is the whole problem, because in the obvious architecture it is quietly consuming the workers that every other customer's deliveries are waiting for. Meanwhile the product's own checkout has to acknowledge a payment in milliseconds, and it cannot be waiting on any of this.

Intake accepts an event, persists it, and acknowledges — in that order, knowing nothing about who is subscribed, so intake latency is independent of subscription count. Fan-out happens afterwards on the platform's own time, turning one event into N independent units of work, each carrying an idempotency key that is stable across every retry and every replay. Each unit lands in its endpoint's own queue, or its own addressable group within one, so that a worker leasing work for one endpoint can never be blocked by another. Delivery workers run in egress-only subnets with no route back into the estate: they re-resolve the address immediately before connecting, sign the timestamp and the raw body under a KMS-wrapped per-endpoint secret, POST through a NAT range the platform publishes as a public interface, and classify the outcome into one of four named classes. Retryable failures back off on a published schedule for 72 hours; permanent ones are never retried and never silent. What is left over is dead-lettered, the owner is told at the first one, and the backlog waits for a replay the customer asks for.

## What it is, and what it is not

- **Per-endpoint queues as the isolation boundary** — not one pending-delivery queue drained by a shared worker pool, which is correct in every steady-state diagram and fails the first time a large customer's endpoint gets slow.
- **Intake acknowledged on durability alone** — not subscription matching inside the product's write path, where adding 500 endpoints makes checkout slower.
- **At-least-once, published, with a stable idempotency key** — not an exactly-once claim the platform cannot keep and the consumer will therefore not defend against.
- **Ordering declared per subscription** — not strict per-endpoint ordering by default, where one poison payload is a 72-hour outage for that customer.
- **A canonical signed string published as a contract** — not a signature scheme described in prose that every consumer implements slightly differently and half of them get wrong.
- **The egress address range treated as a public interface** — not a NAT range that changes with a routine infrastructure ticket and breaks 40,000 allowlists.
- **Network topology as the containment for request forgery** — not an SSRF validation function as the only thing between a stranger's URL and the metadata service.
- **Giving up loudly at the first dead letter** — not an endpoint that silently stops receiving and is discovered a week later by the customer's finance team.

## The decisions that are the architecture

1. **Capture and delivery are separate planes** (ADR-01) — Intake persists and acknowledges without consulting subscription state. Nothing downstream can make a product transaction slower or less available.
2. **The endpoint is the unit of isolation** (ADR-02) — Not the tenant. Two endpoints belonging to the same customer have independent queues, circuits, backlogs and health.
3. **At-least-once, said out loud** (ADR-04) — A stable idempotency key per delivery, constant across retries and replays, and a consumer contract that puts deduplication where it can actually be done.
4. **Ordering is declared, not assumed** (ADR-05) — Strict ordering turns one stuck delivery into a total outage for that endpoint. It is offered, with the consequence stated, rather than given by default.
5. **The signature is a published contract** (ADR-06) — Canonical string, two active secrets, seven-day rotation overlap, and worked verification examples. A signature nobody can verify is decoration.
6. **The retry schedule belongs to the platform** (ADR-08) — Twelve attempts over 72 hours with full jitter. Customer-configurable retries make capacity unplannable and let a tenant build a self-inflicted outage.
7. **Give up loudly** (ADR-09) — Notification at the first dead letter, not the hundredth, through a channel that does not depend on the endpoint working.
8. **The network contains the worker, not the guard code** (ADR-12) — Egress-only subnets with no route inward. The address check is a control; the topology is the control that holds when the check has a bug.
9. **Two systems of record** (ADR-15) — What we were asked to send, and what we actually did. Neither is derivable from the other, and the queues are rebuilt from the second.

## Why this should still be right in ten years

A webhook platform outlives the product it was built for, because every customer integration depends on the shape of the request and the semantics of the retry. These are the properties that should survive a change of scale, of cloud, and of the people who built it.

- **The capture/delivery boundary depends on no technology.** ADR-01 says the product's write path ends at durability. That survives replacing SQS, Fargate, DynamoDB, AWS and the language everything is written in. What would break it is a future team adding a 'synchronous webhook' option for a customer who asked nicely — which is why it is drawn as a boundary in view 11 rather than left as a convention.
- **The idempotency key is the contract that ages best.** Every other guarantee on this platform can be tightened or loosened. The key cannot, because consumers have written deduplication against it and a change to its stability silently double-processes payments. It is the one field where the right answer in ten years is exactly the answer today.
- **Isolation granularity is a property, not an implementation.** ADR-03 will be revisited — SQS queue limits, cost per queue and the traffic distribution will all change, and the hybrid promotion scheme may be replaced entirely. ADR-02's claim, that two endpoints never share a fate, is what must survive that rewrite, and it is stated separately for exactly that reason.
- **The egress range outlives the architecture.** Of everything here, the published NAT range is the hardest thing to change, because it lives in 40,000 firewall rules the platform cannot see. Treating it as a versioned public interface from day one (ADR-11) is the decision a future team will be most grateful for and least likely to have made themselves.
- **Retry semantics are a promise, not a setting.** Twelve attempts over 72 hours will be argued about forever. What must not drift is that the schedule is published, uniform and the platform's — because the moment it becomes per-tenant configuration, capacity planning, support answers and the meaning of 'we tried' all stop being sayable in one sentence.

## Non-functional targets

Every number here is a stated assumption from the requirement, chosen to be argued with. The right-hand column names the view where the mechanism that meets it is drawn.

| Quality | Target | How it is met | View |
|---|---|---|---|
| Intake availability | ≥ 99.99% monthly | Stateless intake behind a load balancer across three AZs; acknowledgement depends only on the durable write, never on subscription state. | 11 |
| Intake latency | ≤ 15 ms p99, independent of subscription count | Persist and acknowledge; matching and fan-out happen after the response. | 11 |
| Delivery-plane availability | ≥ 99.9% monthly | Measured as the share of deliveries dispatched inside the latency objective to healthy endpoints — a definition that excludes the consumer's own failures. | 17 |
| Time to first attempt | p50 ≤ 250 ms · p99 ≤ 2 s from acceptance | Fan-out and enqueue are asynchronous but not batched; workers lease continuously rather than on a poll interval. | 11 |
| Throughput | 25,000 events/s steady · 120,000/s peak · 45,000 attempts/s | Workers scale horizontally with no shared coordination point whose capacity depends on endpoint count. | 15 |
| Delivery success | ≥ 99.5% first attempt · ≥ 99.95% within three, to healthy endpoints | Per-attempt timeouts sized against a synchronous consumer handler; full-jittered backoff to avoid synchronised retries. | 12 |
| Isolation | No endpoint's backlog delays any other, including same-tenant | Per-endpoint queue or message group, per-endpoint concurrency cap, per-endpoint circuit. | 07 |
| Durability of accepted events | RPO 0 in-region · ≤ 5 s cross-region | Acknowledged only after the durable write; DynamoDB global tables and S3 cross-region replication. | 15 |
| Recovery | Delivery RTO ≤ 15 min · intake RTO ≤ 5 min | Warm standby with state pre-replicated and a second egress range already published. | 15 |
| Subscription propagation | change ≤ 30 s p99 · unsubscribe ≤ 5 s p99 | Strongly consistent read at fan-out; unsubscribes fail closed rather than serving a stale copy. | 20 |
| Secret rotation | effective ≤ 60 s p99 · 7-day two-secret overlap | Both signatures sent during the overlap; worker key cache TTL bounded below the propagation target. | 13 |
| Retention | payloads 30 d · attempts 90 d · dead letters 30 d · audit 400 d | Enforced by storage lifecycle and TTL attributes rather than by an application deletion job that can stall. | 09 |
| Cost | ≤ $0.40 per million deliveries all-in | Retries accounted separately from first attempts; alarm when retries exceed 25% of total attempts. | 17 |

## Scope

**In scope**

- Endpoint registration, event-type subscription, per-endpoint signing secrets and their rotation.
- Durable event intake, fan-out with entitlement evaluation, and the per-delivery idempotency key.
- Delivery execution: timeouts, outcome classification, retry and backoff, circuit breaking and concurrency control.
- Dead-lettering, owner notification, and self-service replay from the console and the API.
- Delivery history, the request inspector and the customer-facing health API.
- The egress path: address validation, redirect handling, the published NAT range and its change process.

**Explicitly out of scope**

- Event production. Product services own their outbox; this platform begins at durable acceptance.
- What a consumer does after returning 200 OK. A 2xx that was not processed is invisible here.
- Internal consumers. A service inside the product reads the event bus directly rather than registering a webhook.
- The customer's own secret storage, firewall configuration and receiver reliability.
- Billing and rating. Delivery counts are attributed per endpoint and handed over.
- Product authorisation. This platform evaluates whether an endpoint is entitled to an event type; what a scope permits elsewhere is the product's.

## What a four-week prototype should prove

Three of this architecture's claims are cheap to test and expensive to be wrong about. A prototype that proves these can be built on; one that skips them is a demonstration of sending an HTTP request, which is the easy half.

1. Isolation under a slow consumer: one endpoint that accepts connections and responds in 14.9 seconds, at the endpoint's full concurrency cap, with the p99 time-to-first-attempt measured for every other endpoint in the same fleet throughout.
2. The queue mapping at population scale: 40,000 endpoints mapped onto whichever scheme ADR-03 chooses, with the per-queue cost, the service-limit headroom and the tail-endpoint latency measured — this is the number that decides whether the hybrid in Phase 2 is a nicety or the only workable answer.
3. The recovery ramp: an endpoint down for four hours with a full backlog, brought back, with the load it receives measured against its own capacity over the first five minutes. The failure mode being tested is the platform causing the customer's second outage.
4. Verification across languages: the published canonical string implemented independently in the three most common customer stacks, against a body containing Unicode, a trailing newline and a number that re-serialises differently — the three things that actually cause signature mismatches.

- Kill the subscription store mid-fan-out and confirm delivery continues on last known state while an unsubscribe is refused rather than silently served from a stale copy.
- Delete every per-endpoint queue and confirm the backlog is rebuilt from the attempt log with nothing already recorded as delivered re-sent.
- Point an endpoint at a host whose DNS flips to a private address between registration and delivery, and confirm the attempt is rejected before dispatch and recorded as such.
- Rotate a signing secret mid-flight and confirm no delivery fails verification at any point during the overlap.

## Open risks, carried rather than hidden

| Risk | If it lands | Response |
|---|---|---|
| The queue-per-endpoint model does not survive the population | 40,000 SQS queues hits a service limit or a cost line, and the fallback to shared queues with message groups gives ordering isolation without throughput isolation — which is not the promise in ADR-02. | Measure in the prototype. The designed fallback is the hybrid in ADR-03: dedicated queues for the hot tail, shared groups for the long tail, with automatic promotion. |
| Strict ordering is demanded and then regretted | A large customer asks for per-endpoint ordering, gets it, and discovers the first poison payload blocks their stream for 72 hours. The platform is blamed for honouring the request. | Ordering is a declared per-subscription property with its consequence stated at subscription time, and per-resource ordering in Phase 2 confines the blocking to one entity rather than the whole endpoint. |
| Auto-disable fires during a customer's own incident | The platform unilaterally breaks a working configuration at the worst possible moment, and the notification is missed because everyone is busy. | The backlog survives the disable, re-enable offers an explicit choice, and the first-dead-letter notification precedes the disable by up to 72 hours. If notification reliability cannot be demonstrated, the disable threshold is the number that should move. |
| The egress range has to change | 40,000 customer firewall rules are wrong and the platform cannot see any of them. Deliveries fail as connection timeouts, which look identical to the customer being down. | Both regions' ranges are published before launch, the range is over-provisioned, and any change follows a published deprecation window with dual-range operation. |
| Retry traffic dominates capacity | A population of permanently failing endpoints consumes a majority of delivery workers while producing no successful deliveries, and the cost per useful delivery quietly triples. | Retries are accounted separately, alarmed above 25% of attempts, and per-endpoint retry cost is surfaced so a pathological endpoint is identifiable rather than absorbed. |
| Fat payloads become the dominant cost | At 25,000 events/s, a 30-day retention and a p99 payload above the assumed 64 KB, payload storage outgrows everything else on the platform. | Payload stored once per event rather than per delivery; retention is a per-plan lever; the thin-event option in ADR-04's context remains available if the numbers force it. |

The reasoning behind every component and technology choice is in the [Architecture Decision Record](decision-record): 16 records across 6 areas, each with the alternatives that lost and what the choice costs.