Architecture Decision Record

Webhook Delivery Service · Solution Architecture v1.0 · Amazon Web Services · Integration Platform Architecture · 2026-09 · 20 views · 16 architecture decision records

The argument these decisions serve is summarised in the Architecture One-Pager.

Sixteen decisions make up this architecture. Everything else across the twenty views is convention, sizing or consequence. Each record states the forcing question, the context that makes it hard, what was decided, how it is realised on Amazon Web Services, the options weighed, what the choice buys and costs, the conditions that would flip it, why it should still be right in ten years, and the lesson worth carrying to a different system.

Status of this document. This is a design, not a report on a running system. The rates, latencies, volumes, retentions and windows are the requirement's stated assumptions for a mid-size B2B SaaS product with an active developer ecosystem — 8,000 tenants, 40,000 registered endpoints, 25,000 accepted events per second, 45,000 delivery attempts per second — invented to be defensible and arguable rather than absent. They are to be replaced by measured telemetry before build, and three of them in particular are the first numbers real data should overturn: the 12% retry share, which decides how much of the fleet is spent producing nothing; the p99 fan-out of 40, which decides whether the queue mapping in ADR-03 survives; and the 30-day payload retention, which silently defines the replay window every customer will eventually discover the hard way.

How to read a record

• Question: The forcing question: why a decision was needed at all.
• Context: The requirement, the scale and the constraint that make it hard.
• Decision: What this architecture does, stated so it can be checked.
• How it is realised on Google Cloud: The concrete mechanism: which service or package, configured how, in which project.
• Options weighed: Chosen, rejected, deferred, or right elsewhere, with the reason for each.
• Consequences: What the choice buys and what it costs, both kept visible.
• Choose differently when: The conditions that would flip the decision for your system.
• Why it holds up over time: What keeps the decision right as scale, staff and technology change.
• Lesson: The principle that transfers beyond this platform.

Decision map

Planes and isolation: Where the product's involvement ends, and what may never share a fate with what.

• ADR-01 · Capture and delivery are separate planes joined by a durable log
• ADR-02 · The endpoint, not the tenant, is the unit of isolation
• ADR-03 · Endpoints map onto physical queues by a declared scheme, starting with message groups

The delivery contract: What the platform promises a consumer, in words a consumer can build against.

• ADR-04 · At-least-once, published, with an idempotency key stable across retries and replays
• ADR-05 · Ordering is declared per subscription, and defaults to none

Authenticity: How a consumer knows a request came from here, and how a secret changes without a cut-over.

• ADR-06 · HMAC over a published canonical string, with asymmetric signatures deferred
• ADR-07 · Two active secrets with a seven-day overlap, rotated by the customer at their own pace

Failure handling: What happens when the other end is slow, broken, hostile, or simply gone.

• ADR-08 · The retry schedule belongs to the platform, and resumption is ramped
• ADR-09 · Give up loudly: notification at the first dead letter, on a channel independent of the endpoint
• ADR-10 · Replay is a new delivery with the original key, rate-capped and confirmed

The egress path: Pointing a worker fleet at URLs strangers chose, safely and attributably.

• ADR-11 · The egress address range is a public interface with a change process
• ADR-12 · Network topology, not validation code, is the containment for request forgery

Evidence and control: What is recorded, who may change what, and how a developer diagnoses their own integration.

• ADR-13 · Failure is self-diagnosable, and platform time is measured separately from consumer time
• ADR-14 · Auto-disable after 72 hours, with the backlog retained and re-enable an explicit choice
• ADR-15 · Two systems of record: what we were asked to send, and what we actually did
• ADR-16 · Subscription reads are strongly consistent at fan-out, and a URL change is an elevated action

Technology by capability

Every capability and what it is realised with, the origin of that choice, the alternative that was weighed, and the record that argues it. The requirement stays vendor-neutral throughout; this table is where the architecture commits.

The decisions, and the alternatives that lost

Planes and isolation

Where the product's involvement ends, and what may never share a fate with what.

ADR-01 · Capture and delivery are separate planes joined by a durable log

Status: Accepted · Shown on views: 02, 06, 11

Does the product's write path know anything about who is subscribed, or does it hand an event to a durable store and stop?

Context. The naive integration is the obvious one: when an order is placed, look up the subscriptions for order.created, and post to each. It works perfectly in development and it makes checkout latency a function of how many customers have subscribed, how many of them are slow today, and whether the subscription store is healthy. Every variant that keeps matching in the write path has the same defect in a smaller form — even a fast in-memory match makes a product transaction depend on a component whose whole job is to deal with the public internet. Meanwhile the product needs to acknowledge a payment in milliseconds and be right about it.

Decision. Intake validates, assigns an event id, persists the payload durably, and acknowledges — in that order — without consulting subscription state, endpoint health, or any external network. Subscription matching, entitlement evaluation and fan-out happen after the acknowledgement, on the platform's own time. An acknowledged event will be delivered or dead-lettered; it will never be lost. Producers hold events in their own transactional outbox, so an intake outage degrades the product briefly and visibly rather than silently dropping events.

How it is realised on Google Cloud. Intake is a stateless Fargate service behind an ALB across three AZs. The payload goes to S3 under an event id and the index row to DynamoDB in the same request path; the 202 is returned once both are durable. Fan-out consumes the committed-event stream asynchronously. The intake service holds no reference to the subscription table at all — the dependency does not exist rather than being unused.

What it buys

• Intake latency is independent of subscription count, so a tenant adding 500 endpoints cannot slow down checkout.
• An intake outage degrades the product visibly and briefly, rather than quietly losing events.
• The delivery plane can be scaled, deployed and broken independently of anything the product does.

What it costs

• One extra hop before the first attempt, which is why time-to-first-attempt carries its own p99 objective.
• The platform now owns durability for events it has acknowledged, which is what forces the RPO 0 requirement and the cross-region replication.

Choose differently when. Match in the write path when there is exactly one consumer, it is internal, and the write is already slow enough that one more call does not matter. A batch export to a single partner is in that category; a public webhook product never is.

Why it holds up over time. This is the decision least tied to any technology on this page and most likely to be eroded by a future feature request. Nothing about it depends on SQS, Fargate or AWS. What would break it is a 'synchronous webhook' option sold to one large customer, which is why the boundary is drawn explicitly in view 11 rather than implied.

Lesson. The write path is the one place in a system where somebody else's uptime must never appear. Decide where it ends before anyone asks for an exception.

ADR-02 · The endpoint, not the tenant, is the unit of isolation

Status: Accepted · Shown on views: 07, 12, 18

When one consumer misbehaves, what is the smallest thing that is allowed to be affected?

Context. Multi-tenant platforms reflexively isolate by tenant, and here that is not enough. A tenant with five endpoints — production, staging, an analytics sink, a partner relay and one somebody forgot — will have one of them broken at any given time, and the other four must keep delivering. The failure being designed against is specific: an endpoint that accepts the connection and responds in fifteen seconds without ever erroring. Shared workers fill with in-flight attempts against that one endpoint, and the platform's availability becomes a function of the least reliable server in its customer base. A per-endpoint concurrency cap mitigates this and does not fix it, because a cap on in-flight attempts does not stop that endpoint's messages occupying the head of a shared queue while its circuit is open for six hours.

Decision. The endpoint is the unit of registration, configuration, queueing, concurrency, circuit breaking, health, backlog, accounting and failure. Two endpoints belonging to the same tenant are as isolated from each other as two endpoints belonging to different tenants. A worker leasing work for one endpoint can never be blocked by another endpoint's backlog. Per-tenant controls exist — a fan-out budget, an endpoint quota — but they are budgets, not the isolation boundary.

How it is realised on Google Cloud. Each endpoint has its own SQS FIFO message group (and, for the hot tail, its own queue — see ADR-03), its own concurrency cap enforced by the worker leasing logic, its own circuit state, its own dead-letter partition and its own attempt-log partition key. The delivery history API and every alert are scoped to an endpoint id.

What it buys

• A slow consumer's blast radius is exactly one endpoint, which is the only defensible answer to give a customer.
• Every operational number — backlog, success rate, retry share, cost — has a natural owner.
• Circuit breaking becomes safe to be aggressive about, because opening a circuit harms nobody else.

What it costs

• Cardinality of 40,000 across queues, circuits, health records and partitions, which is a real infrastructure constraint rather than a bookkeeping one.
• Fair scheduling across endpoints becomes a requirement, since arrival order no longer produces a sensible allocation.

Choose differently when. Isolate by tenant when tenants have one endpoint each, or when the consumer population is small and operationally competent — an internal event bus between a dozen teams does not need 40,000 of anything.

Why it holds up over time. The claim survives every plausible change of implementation. ADR-03 will be revisited when SQS limits, per-queue cost or the traffic distribution change; this record states the property that rewrite must preserve, which is why the two are separate decisions rather than one.

Lesson. Isolate at the granularity of the thing that actually fails. Tenancy is an accounting boundary, and accounting boundaries make poor blast-radius boundaries.

ADR-03 · Endpoints map onto physical queues by a declared scheme, starting with message groups

Status: Accepted · Shown on views: 07, 09, 15

40,000 endpoints, each needing its own isolated stream — how many physical queues is that, and what breaks first?

Context. ADR-02 demands per-endpoint isolation without saying how it is realised, and the two obvious realisations fail differently. A queue per endpoint gives perfect isolation and a trivial mental model, and 40,000 queues is a management-plane problem, a per-queue cost line, and in most clouds a hard service limit that requires a quota conversation. One shared queue with a message group key per endpoint gives ordering isolation for free and throughput isolation not at all: a hot endpoint's messages still occupy the shared queue's consumers. The traffic distribution is the deciding fact and it is not known before launch — a webhook population is invariably long-tailed, with a handful of endpoints carrying most of the volume.

Decision. The mapping from endpoints to physical queues is an explicit, versioned scheme rather than an emergent property. The MVP uses a bounded set of shared SQS FIFO queues with the endpoint id as the message group key, giving ordering isolation and per-endpoint lease fairness, plus per-endpoint concurrency caps. Phase 2 adds automatic promotion: an endpoint whose volume or backlog crosses a threshold is moved to a dedicated queue, and demoted when it falls back. The promotion control loop is accepted as work rather than avoided.

How it is realised on Google Cloud. SQS FIFO queues with MessageGroupId = endpoint_id. Workers lease across groups with a fairness policy rather than draining a single group, so a large backlog in one group does not monopolise a worker. Promotion writes a queue assignment into the endpoint record; the planner reads it at enqueue time, and a drain window ensures no in-flight message is orphaned by a move.

What it buys

• The MVP ships without a 40,000-queue quota conversation, and the promotion scheme has measured traffic to be designed against.
• Ordering isolation is available immediately, since the group key is the natural unit.

What it costs

• Throughput isolation in the MVP is bounded by lease fairness rather than by physical separation, which is a weaker promise than ADR-02 makes and is stated as such.
• Phase 2 adds a promotion/demotion control loop, including the drain window that stops a move orphaning in-flight work.

Choose differently when. Go queue-per-endpoint immediately when the endpoint population is in the low thousands, or when the queueing technology makes a queue nearly free. Stay with shared groups permanently when the traffic distribution turns out to be flat, which would be surprising and worth knowing.

Why it holds up over time. This is the decision on the page most likely to be replaced, and that is by design: it is the implementation of ADR-02 rather than a principle of its own. A future team should feel free to rewrite it entirely, provided the property in ADR-02 survives the rewrite.

Lesson. When a principle needs an expensive implementation, write them down as two decisions. The principle should outlive the mechanism, and it will not if they are in the same record.

The delivery contract

What the platform promises a consumer, in words a consumer can build against.

ADR-04 · At-least-once, published, with an idempotency key stable across retries and replays

Status: Accepted · Shown on views: 10, 11, 14

What exactly is promised to a consumer about how many times they will see an event, and who is responsible for the difference?

Context. There is no way to deliver an HTTP request exactly once. The consumer can process an event and have the response lost on the way back; the platform then cannot distinguish that from a consumer that never received it, and must choose between retrying (duplicate) and not retrying (loss). Every webhook platform makes this choice, and the ones that describe their behaviour vaguely end up with consumers who did not implement deduplication because nobody told them clearly that they had to. The failure is silent and it happens in the consumer's ledger.

Decision. At-least-once is the published guarantee, stated plainly rather than implied. Every delivery carries an idempotency key that is stable across every retry of that delivery and across every replay of it, and the consumer documentation requires deduplication on that key. The platform prefers a duplicate to a loss in every ambiguous case and says so. A 2xx response that the consumer did not actually process is outside the platform's power to detect and is named as such rather than glossed over.

How it is realised on Google Cloud. The key is minted at fan-out and stored on the delivery row, sent as a header on every attempt, and reused verbatim by the replay service. The documented contract names the header, states the retry schedule, and says in one sentence that a consumer must be safe to receive the same key twice.

What it buys

• Replay becomes safe without a new mechanism: it reuses the key the consumer is already deduplicating on.
• Regional failover can produce duplicates without that being an incident, because duplicates are inside the contract.
• The platform can be aggressive about retrying, which is what makes the 72-hour window valuable.

What it costs

• Every consumer must implement deduplication, and some will not. The mitigation is documentation and examples, not a stronger guarantee.
• The unclosable gap — consumer processed it, response lost — remains, and shows in the attempt log as a retryable failure that was actually a success.

Choose differently when. Move to at-most-once only where duplicate processing is catastrophic and loss is not, which is rare and usually indicates the event should not have been a webhook.

Why it holds up over time. The idempotency key is the field on this platform that can never change. Consumers write deduplication against it, and a change to its stability silently double-processes their transactions years later. Everything else here can be tightened; this cannot.

Lesson. State the weak guarantee loudly rather than the strong guarantee quietly. A consumer who knows they must deduplicate is safer than one who was told they did not have to.

ADR-05 · Ordering is declared per subscription, and defaults to none

Status: Accepted · Shown on views: 10, 12

Are events delivered to an endpoint in order, and if so, in order of what?

Context. Consumers ask for strict per-endpoint ordering because it is obviously easier to reason about. It is also what makes one stuck delivery block everything behind it: a payload the consumer can never accept, sitting at the head of an ordered stream, is a 72-hour total outage for that endpoint rather than one dead letter. Per-resource ordering — all events about one order, one subscription, one pull request — confines the blocking to a single entity, at the cost of multiplying the number of ordered groups by orders of magnitude, well past what a queueing layer will address. No ordering at all is cheapest and pushes reconciliation onto the consumer, who will not do it and will then report a bug.

Decision. Ordering is a declared property of a subscription, with three modes and an explicit consequence attached to each: none (the default, events may arrive out of order and carry a timestamp for reconciliation), per-endpoint (strict, with head-of-line blocking stated at subscription time), and per-resource (Phase 2, ordered within a resource id). 'No ordering guarantee' is a legitimate, documented choice rather than an admission.

How it is realised on Google Cloud. The mode lives on the endpoint record. None and per-endpoint both use the endpoint id as the SQS FIFO message group key, differing in whether a blocked message halts the group or is moved aside; per-resource uses a composite group key and requires the queue mapping of ADR-03 to be revisited first.

What it buys

• The customer who wants strict ordering can have it, knowing what it costs them.
• The default does not impose a 72-hour failure mode on customers who never asked for ordering.
• The timestamp on every event makes reconciliation possible for the default mode rather than theoretical.

What it costs

• Three delivery behaviours to implement, test and support instead of one.
• A customer on strict ordering who hits a poison payload has a bad day, and the platform's answer is a dead letter and an explanation rather than a fix.

Choose differently when. Offer only strict ordering where the event stream is a state-machine log the consumer replays — a change-data-capture feed, for instance — and only none where every event is independent and self-describing.

Why it holds up over time. The three modes will outlast their implementation. What could break this is a future team making per-endpoint ordering the default because it is what a vocal customer asked for, without carrying the head-of-line consequence into the default.

Lesson. When a guarantee has an ugly consequence, make the consequence part of choosing it. A default that hides a 72-hour failure mode is a trap with good intentions.

Authenticity

How a consumer knows a request came from here, and how a secret changes without a cut-over.

ADR-06 · HMAC over a published canonical string, with asymmetric signatures deferred

Status: Accepted · Shown on views: 08, 13, 16

How does a consumer know a request came from this platform, and what happens to that assurance if the platform's own store is breached?

Context. A shared HMAC secret is what every consumer already knows how to verify, needs no key distribution beyond showing the secret once, and means the platform and the consumer both hold material that can forge a delivery — so a breach of the endpoint store is a forgery capability against every customer at once. An asymmetric signature over a published key removes the secret from the consumer entirely and makes rotation a publish rather than a coordinated change, at the cost of asking every consumer to fetch, cache and verify against a key, which is more code and one more thing to get wrong. The deciding fact is that the population includes people integrating in an afternoon, and a signature they cannot verify is a signature they will skip.

Decision. HMAC-SHA-256 over a canonical string of timestamp plus the exact raw request body, under a per-endpoint secret, with the canonical string published as a contract rather than described in prose, and worked verification examples in the languages the customer base actually uses. The timestamp is inside the signed material so a captured request cannot be replayed indefinitely, and a recommended acceptance window is published. Asymmetric signatures are Phase 3, offered alongside rather than replacing HMAC.

How it is realised on Google Cloud. The per-endpoint secret is generated as a KMS data key, stored wrapped, and shown to the customer exactly once. Workers decrypt with a short-lived cache and sign each attempt over the raw bytes being sent. Both active secrets produce a signature during a rotation overlap (ADR-07). The canonical string is frozen by a contract test in CI, because changing it silently breaks every consumer's verification at once.

What it buys

• Verification is a handful of lines in any language, which is what makes it actually happen.
• Rotation is possible without contacting the consumer, because two secrets are live at once.
• The canonical string being a frozen contract means a refactor of the request builder cannot silently break 40,000 integrations.

What it costs

• A breach of the endpoint store is a forgery capability against every customer, mitigated only by wrapping the secrets under KMS and by the Phase 3 asymmetric option.
• The platform cannot contain a secret compromise without the customer's cooperation, since only they can stop trusting the old signature.

Choose differently when. Go asymmetric first when the consumer population is sophisticated, when the platform's own compromise must be containable unilaterally, or when a regulator requires non-repudiation the platform itself cannot produce.

Why it holds up over time. The canonical string, not the algorithm, is the thing that must survive. SHA-256 will be replaced; the rule that the signature covers the timestamp and the exact raw bytes, and that the string is published rather than described, should not be. Signing a re-serialised body is the defect that would quietly break everything, which is why view 13 shows the verifier recomputing over raw bytes.

Lesson. A signature nobody can verify is decoration. The documentation, the examples and the frozen canonical string are part of the security control, not support material around it.

ADR-07 · Two active secrets with a seven-day overlap, rotated by the customer at their own pace

Status: Accepted · Shown on views: 10, 13

How does a signing secret change without a coordinated cut-over between the platform and a consumer who may be asleep?

Context. A single active secret makes rotation an atomic, coordinated event: the platform switches, and every request between that moment and the consumer's own deploy fails verification. For a consumer who rotates during their own business hours in a different timezone, that window is hours of dead letters. The failure is also the worst kind, because it looks exactly like an attack: valid-looking requests failing signature verification.

Decision. Each endpoint may have two active secrets, bounded by active_from and active_until. During an overlap the platform sends a signature for each, and the consumer accepts a request if either verifies. The customer rolls a secret whenever they like, migrates their code at their own pace within a seven-day overlap, and retires the old one when they are ready. A secret is never displayed again after creation.

How it is realised on Google Cloud. Both secrets live on the endpoint record as KMS-wrapped data keys with validity windows. The worker reads both, signs twice, and sends both signature values in one header. Rotation is a data change, effective within the propagation bound, not an operational procedure requiring coordination.

What it buys

• A leaked secret can be rotated immediately without the platform having to coordinate anything with the customer.
• Rotation produces no failed deliveries, so customers actually do it.
• The overlap window is a data property that can be shortened per endpoint during an incident.

What it costs

• Two HMAC computations per attempt at 45,000 attempts/s, which is measurable but small relative to the network cost.
• A straggler who never migrates keeps a compromised secret valid until active_until, which is why the window is bounded rather than open-ended.

Choose differently when. A single secret with an atomic switch is fine where the platform and the consumer are the same organisation and can deploy together. Beyond that boundary, overlap is not optional.

Why it holds up over time. The overlap mechanism survives a change of algorithm and a change to asymmetric keys, where it becomes two published keys rather than two secrets. The seven days is a parameter; the two-live-credentials shape is the decision.

Lesson. Any credential shared across an organisational boundary needs two of it to be live at once. A rotation that requires both sides to act at the same moment is a rotation that will not happen.

Failure handling

What happens when the other end is slow, broken, hostile, or simply gone.

ADR-08 · The retry schedule belongs to the platform, and resumption is ramped

Status: Accepted · Shown on views: 05, 12, 14

Who decides how often a failing endpoint is retried, and what happens the moment it comes back?

Context. Per-subscription retry configuration is genuinely useful: a consumer who knows their maintenance window is an hour wants different behaviour from one running a serverless handler that cold-starts. It also lets a customer configure a schedule that is a self-inflicted denial of service, makes the platform's retry capacity unpredictable at 45,000 attempts per second, and turns the support answer to 'when will you retry?' into 'it depends on what you set'. The second half of the problem is the opposite: when a large consumer returns after four hours down, draining their backlog at full rate is the platform causing that customer's second outage.

Decision. One published schedule: twelve attempts over 72 hours, full-jittered exponential backoff from a five-second base, capped at six hours between attempts. Retry-After on 429 and 503 is honoured up to a one-hour ceiling, because a consumer telling the platform when to come back is the most useful signal it will ever get. A circuit opens after twenty consecutive failures and probes once every five minutes with a single attempt. On recovery, resumption ramps from 10% to 100% of the endpoint's concurrency cap over five minutes. Named retry profiles, rather than free configuration, are Phase 2.

How it is realised on Google Cloud. Backoff is computed per delivery and written to next_attempt_at; SQS visibility timeout carries the short intervals and a scheduler re-enqueues the long ones. Full jitter rather than fixed exponential specifically to avoid a synchronised retry wave when a popular consumer returns. The ramp is enforced by the worker's per-endpoint concurrency governor, not by the queue.

What it buys

• Retry capacity is plannable, and 'when will you retry?' has a one-sentence answer.
• Full jitter means a popular consumer's recovery does not produce a thundering herd from the platform.
• The ramp means the platform is never the cause of a consumer's second outage.

What it costs

• A consumer whose operational reality does not fit the schedule has no recourse until Phase 2.
• 72 hours of retries for a permanently dead endpoint is capacity spent producing nothing, which is what the auto-disable in ADR-14 bounds.

Choose differently when. Allow free configuration where consumers are internal, few, and accountable for the capacity they consume. On a public platform with 40,000 endpoints, the schedule is infrastructure.

Why it holds up over time. The numbers will be argued about forever and should be. What must not drift is that the schedule is published, uniform and the platform's — the moment it becomes per-tenant configuration, capacity planning and support both stop being sayable in one sentence.

Lesson. Retry behaviour is a promise, not a setting. And whatever the schedule, ramp the recovery: the system that just came back is the one least able to take what you have been saving up for it.

ADR-09 · Give up loudly: notification at the first dead letter, on a channel independent of the endpoint

Status: Accepted · Shown on views: 05, 12, 18

When a delivery finally fails for good, who finds out, and when?

Context. The worst outcome a webhook platform can produce is an integration that stops working and nobody notices — discovered a week later by a customer's finance team reconciling numbers that do not add up. The temptation is to notify on the state change that feels most significant, which is the endpoint being disabled after 72 hours. By then the customer has lost three days of events and the platform's message arrives as an accusation rather than a warning. The first dead letter is the actionable moment; the hundredth is noise.

Decision. The tenant is notified when deliveries begin dead-lettering, not when the endpoint is eventually disabled. A permanent failure — a 4xx that is not 408 or 429 — is never retried and never silent: its owner is told, because a misconfigured path or a revoked credential produces permanent failures indefinitely and produces them quietly. Notification goes through a channel that does not depend on the customer's endpoint being healthy. Subsequent dead letters within a window are aggregated rather than each producing a message.

How it is realised on Google Cloud. The first dead letter for an endpoint within a rolling window raises an event to the product's existing notification service, which reaches the same people who receive every other account notification. The endpoint's health state moves to degraded or failing and becomes visible in the console and in the delivery health API before any notification is sent, so a customer watching their own dashboard sees it first.

What it buys

• The customer learns at the earliest actionable moment, on a channel that works when their endpoint does not.
• A permanent failure has an owner instead of accumulating silently.
• Notification volume is bounded, so the channel stays credible.

What it costs

• A dependency on a notification service outside the delivery boundary, accepted because it is not on the delivery path.
• Occasional false alarms from endpoints that dead-letter once and recover, which is the correct side to err on.

Choose differently when. Suppress notification entirely where every consumer has been contractually required to monitor delivery health themselves, and where the platform has a way to verify they do.

Why it holds up over time. Independent of every technology here. The failure it prevents — silent, discovered late, discovered by someone else — is the same failure in any asynchronous integration, and the instinct to notify on the dramatic state change rather than the early one is equally durable.

Lesson. Alert on the first occurrence of a thing that can be fixed, not on the state change that feels most serious. The two are rarely the same moment.

ADR-10 · Replay is a new delivery with the original key, rate-capped and confirmed

Status: Accepted · Shown on views: 12, 14, 17

How does a customer get back the events they missed, without that recovery being its own outage?

Context. A consumer down for four days has a backlog waiting for them. Handing it over is the point of retaining it, and handing it over at full speed does to their system exactly what the original outage did — with the added difficulty that they asked for it, so it looks like their fault and feels like the platform's. There is also a request that always arrives and sounds unreasonable until you think about it: 'we processed those events and then lost our database, can you send them again?' That is not a failure of this platform, and refusing it is refusing the only copy of data the customer has.

Decision. A replay is a new delivery carrying the original event id and the original idempotency key, so the consumer's existing deduplication still protects them. Replay of successfully delivered events within the retention window is supported, not only of dead letters. Replay is rate-capped per endpoint, requires explicit confirmation above a declared size, and runs on a separate worker pool so it cannot starve first-attempt delivery. A dead letter is never deleted before its retention expires, including when the endpoint is deleted, unless the tenant explicitly purges it.

How it is realised on Google Cloud. The replay service reads the dead-letter index or the delivery records, re-enqueues to the endpoint's message group with the original key, and writes replayed_as on the original so the relationship is traceable. Replay workers are a distinct Fargate service with their own concurrency budget. The bulk-replay confirmation is a product surface, not a rate limit — the customer is being told what they are about to do.

What it buys

• Self-service recovery with no support ticket and no new safety mechanism.
• A bulk replay is survivable for the consumer, and traceable afterwards through replayed_as.
• Replay cannot degrade first-attempt delivery for anyone, including the tenant replaying.

What it costs

• A separate worker pool that is idle most of the time.
• The retention window becomes a hard customer-facing contract, because it silently defines what can be replayed at all.

Choose differently when. Replay at full rate where the consumer is internal, their capacity is known, and the platform can coordinate the drain with them directly.

Why it holds up over time. Replay's safety comes entirely from ADR-04's stable key. As long as that holds, replay can be reimplemented freely. What would break it is a future 'replay with a fresh id' option added to work around a consumer who deduplicates incorrectly, which would fix one customer and silently endanger every other.

Lesson. Recovery mechanisms deserve the same rate limiting as the traffic they are recovering from. The customer asking for a flood is still the customer who gets flooded.

The egress path

Pointing a worker fleet at URLs strangers chose, safely and attributably.

ADR-11 · The egress address range is a public interface with a change process

Status: Accepted · Shown on views: 08, 15, 16

Where do deliveries appear to come from, and what does it take to change that?

Context. A meaningful share of enterprise consumers will only accept inbound traffic from allowlisted source addresses. Publishing the range is therefore a product requirement, not an operational convenience. The consequence is usually discovered too late: once published, the range lives in tens of thousands of firewall rules the platform cannot see, cannot query and cannot update. An infrastructure ticket that moves a NAT gateway then presents, to the customer, as the platform being unreachable — and to the platform, as thousands of endpoints simultaneously timing out, which is indistinguishable from a widespread consumer problem.

Decision. The egress range is a versioned, published, documented interface, changed only through a review gate in the release pipeline with a published deprecation window and dual-range operation during it. Both the active and the standby region's ranges are published before launch, so a regional failover is not also an allowlist change for 40,000 consumers. The range is deliberately narrow and over-provisioned so that growth does not force a change.

How it is realised on Google Cloud. Dedicated NAT gateways on a reserved /28 per region, with the range asserted in Terraform and a CI gate that fails a plan changing it without an accompanying interface-change record. The ranges are served from a machine-readable endpoint so consumers can automate their allowlists.

What it buys

• Enterprise consumers can integrate at all.
• A regional failover is a routing change rather than a coordinated allowlist migration across the customer base.
• Delivery traffic is attributable from outside the platform, which helps when a consumer reports unexpected requests.

What it costs

• Reserved address space in two regions, mostly unused.
• Any future need to change the range is genuinely expensive, which is the cost of having made the promise.

Choose differently when. Skip the published range entirely where every consumer is a cloud-hosted service with no ingress allowlist, which is a bet on the customer base that gets harder to unwind every year.

Why it holds up over time. This is the hardest thing on the platform to change and the decision a future team will be most grateful for. It survives a change of cloud only if the replacement can present the same addresses, which is itself an argument for keeping the range narrow.

Lesson. Anything a customer writes into their firewall is an API. Version it, publish it, and put a gate in front of it on the first day, because you cannot retrofit a change process onto something already deployed in ten thousand places.

ADR-12 · Network topology, not validation code, is the containment for request forgery

Status: Accepted · Shown on views: 15, 16, 19

A worker fleet connects to URLs strangers chose. What stops one of those URLs being something inside the estate?

Context. This is the platform's defining security property, and it is unusual: the service is, by design, a machine that makes HTTP requests to arbitrary addresses supplied by unauthenticated third parties. Every classic server-side request forgery technique applies directly — a URL resolving to a private range, DNS rebinding between validation and connection, a redirect to the instance metadata service, a scheme downgrade. An address-validation function handles all of them correctly until the day it has a bug, and the population of ways to express an address is large enough that it eventually will.

Decision. Address validation is implemented thoroughly — resolution checked at registration and immediately before every connect, re-checked after every redirect, redirects capped at three, scheme downgrade refused, signature headers re-signed rather than forwarded to a redirected host — and it is explicitly not the containment. Delivery workers run in egress-only subnets with no route to the capture plane, the control plane, the product VPCs or the instance metadata service. If the guard has a defect, the packet has nowhere to go.

How it is realised on Google Cloud. A dedicated VPC with delivery subnets whose route table contains a default route to the NAT gateway and nothing else; no peering, no transit attachment, no VPC endpoints to internal services, metadata service disabled at the task level. The address guard suite runs on every build as a release gate (view 16), covering rebinding, redirect chains and private-range expression forms.

What it buys

• A defect in the address guard is contained by routing rather than exploited.
• The security review has a structural answer rather than a code-review answer, which is what makes it repeatable.
• Workers hold no credentials that would be useful inside the estate, because they have nowhere to use them.

What it costs

• A separate network to operate, and workers that cannot reach internal services for convenience — every dependency they need must be deliberately provided.
• Debugging from a delivery worker is harder, by design.

Choose differently when. Rely on validation alone only where the workload is not attacker-directed — a crawler over a curated list, for instance. The moment a stranger supplies the address, the network has to be the answer.

Why it holds up over time. The reasoning outlives AWS: whatever the platform runs on, the component that connects to attacker-chosen addresses should be somewhere that a successful redirection reaches nothing. What would erode it is a future feature giving the delivery worker a legitimate reason to call an internal service — at which point the right answer is to move that work out of the worker, not to open the route.

Lesson. When a control has to be perfect, give it somewhere safe to fail. Defence in depth is not two checks of the same kind; it is a second control with a different failure mode.

Evidence and control

What is recorded, who may change what, and how a developer diagnoses their own integration.

ADR-13 · Failure is self-diagnosable, and platform time is measured separately from consumer time

Status: Accepted · Shown on views: 04, 13, 17

When a developer says 'we're not receiving your webhooks', how do they find out why without opening a ticket?

Context. Every integration platform's support load is dominated by one question, and the honest answer is usually 'your server returned a 403 four thousand times'. Without a delivery history the customer can read themselves, answering it requires an engineer to query production, which does not scale past a few hundred integrations. The mirror-image problem is a customer reporting that deliveries are slow, when the latency is entirely their own handler — with one aggregate number, the platform has no way to say so and no way to prove it.

Decision. Every attempt is recorded with its timestamp, duration, response status, error class and a bounded prefix of the response body, and that history is exposed to the endpoint's owner for the retention window, including the exact request that was sent so a developer can replay it locally. Platform time and consumer time are measured and reported as separate quantities at every stage. A per-endpoint dashboard and an equivalent API expose delivery rate, success rate, latency and backlog depth, so a customer can alert on their own integration health rather than discovering a break from their users.

How it is realised on Google Cloud. Attempt records in DynamoDB partitioned by endpoint with a 90-day TTL; the request inspector reconstructs the sent bytes from the stored payload and the recorded headers. Fields the product classifies as sensitive are redacted in the history while the signature remains valid over the bytes actually sent. Spans separate queue time, platform processing and time-on-the-wire.

What it buys

• A developer diagnoses their own integration, which is what makes 40,000 endpoints supportable.
• 'Is it you or us' has a measured answer rather than an argument.
• The request inspector shortens the signature-verification trough in view 04, which is where new integrations actually stall.

What it costs

• Attempt history is the platform's largest write volume and a significant storage line.
• Redaction must be correct, since the history displays data the endpoint's owner may not be entitled to see in full.

Choose differently when. Summary counters are enough where consumers are internal teams with access to the same observability platform, and where a shared dashboard already answers the question.

Why it holds up over time. The latency split is the measurement that ages best: scale changes what the platform looks like, not what it is judged on. A future team with different storage and ten times the traffic will still need to answer whether a slow integration is theirs.

Lesson. Build the interface that answers your most common support question before you have the support question. It is much cheaper than the headcount.

ADR-14 · Auto-disable after 72 hours, with the backlog retained and re-enable an explicit choice

Status: Accepted · Shown on views: 05, 12, 18

What happens to an endpoint that has been failing for three days — and who decides?

Context. An endpoint that has failed continuously for 72 hours is almost always abandoned, and retrying it forever is capacity spent producing nothing. Disabling it is also the platform unilaterally breaking a customer's integration, potentially in the middle of that customer's own extended outage, at a moment when nobody is reading email. Both the action and the inaction have a real cost, and there is no option without one. The secondary question is worse: on re-enable, delivering three days of backlog into a system that has just been repaired can repeat the original failure, and skipping it silently loses data the customer believed was safe.

Decision. An endpoint is auto-disabled after 72 hours of continuous failure with zero successes. Its undelivered backlog is retained for the dead-letter window regardless. The tenant is notified at the first dead letter — up to 72 hours before the disable — on a channel that does not depend on the endpoint. Re-enable is an explicit human action with an explicit choice: deliver the backlog, skip it, or resume from a chosen point. The platform does not make that choice on the customer's behalf.

How it is realised on Google Cloud. Health state transitions are computed from recent attempts (view 18). The disable writes an audit record and a notification; the backlog stays in the dead-letter store under its own retention. Re-enable presents the backlog size and the estimated drain time before the choice is made, because a customer choosing to deliver 400,000 events should be told that first.

What it buys

• Wasted retry capacity is bounded.
• No data is lost by the disable itself, so the decision is reversible for 30 days.
• The customer makes the dangerous choice, informed, at a moment of their choosing.

What it costs

• A customer whose outage exceeds 72 hours has their integration disabled during it, which is a bad experience however well it is handled.
• The whole justification rests on the notification being reliable, which is an assumption until measured.

Choose differently when. Never disable where endpoints are few, owned internally, and the retry cost is negligible. Disable much sooner where the consumer population is untrusted and retry capacity is the binding constraint.

Why it holds up over time. The shape — bound the cost, keep the data, hand the risky decision to whoever understands the consequences — outlives every number in it. The 72 hours is a parameter and should move with evidence; the principle should not.

Lesson. When an automatic action will sometimes be wrong, make it reversible and make its consequence recoverable. Then the argument is about the threshold rather than about the mechanism.

ADR-15 · Two systems of record: what we were asked to send, and what we actually did

Status: Accepted · Shown on views: 06, 09, 10

When a customer says an event never arrived, what does the platform consult, and what can it prove?

Context. The convenient design keeps one store, where a delivery row is mutated as attempts proceed and holds the latest outcome. It answers 'what is the state now' well and everything else badly. A customer dispute is a question about history — was it sent, when, how many times, what did our server say — and a mutated row has thrown that away. Deriving one record from the other is also not possible in either direction: the accepted event does not know how many attempts it took, and the attempt log does not contain the payload.

Decision. The accepted-event store is the system of record for what the platform was asked to send. The attempt log is the system of record for what it actually did. Neither is derived from the other, both are immutable once written, and the queue state is derived from the second — reconstructible after a total loss of the queueing layer, without operator input, without re-sending anything already recorded as delivered.

How it is realised on Google Cloud. Events are written once and never mutated; a correction is a new event. Attempt records are append-only, partitioned by endpoint, with a 90-day TTL for the console view and a 13-month aggregate. Delivery rows carry current state as a projection over the attempt log rather than as independent truth, so a lost delivery row is rebuildable.

What it buys

• A dispute is answered from evidence, with timestamps and response codes, rather than from recollection.
• Queue-layer loss is a rebuild, which is why the queues carry no recovery objective of their own.
• The customer-facing delivery history is a view over the record rather than a parallel copy that can disagree with it.

What it costs

• Append-only writes at attempt volume are the platform's largest write load and a major storage line.
• Two retention schedules to reason about, and they are deliberately different.

Choose differently when. A single mutable row is fine where history is never disputed and the queue is itself durable and authoritative — an internal job runner, for example.

Why it holds up over time. The separation survives any storage technology. What would erode it is a performance optimisation that starts updating the delivery row in place and stops writing an attempt record for a successful first attempt — which is the majority case and therefore exactly the record that would be missing when a customer asks about it.

Lesson. A system that will be argued with needs an append-only account of what it did, kept separately from what it was asked to do. Current state is not evidence.

ADR-16 · Subscription reads are strongly consistent at fan-out, and a URL change is an elevated action

Status: Accepted · Shown on views: 06, 19, 20

How fresh must subscription state be when an event is fanned out, and who is allowed to change where a tenant's data goes?

Context. Two problems share one answer. The first is consistency: an unsubscribe the customer has been told took effect, followed by a delivery, is a data-protection incident rather than a scheduling lag — and continued delivery after a tenant deletion is the worst-consequence failure on this platform. A cached subscription projection is cheap and fast and makes exactly that mistake. The second is authority: changing an endpoint's URL redirects the tenant's data somewhere new, and in most permission models it is the same routine operation as changing which event types are sent. An account takeover would use it, and it would look like ordinary configuration.

Decision. Subscription state is read strongly consistent at fan-out time — once per event rather than once per attempt, which is what makes it affordable. When the subscription store is unavailable for reads, delivery continues on last known state and subscription writes are refused, except unsubscribes and deletions, which fail closed. Changing an endpoint URL requires a higher authority and a fresh authentication than changing its event-type selection, returned as a machine-readable step-up rather than a generic denial. Every subscription change, secret rotation, enable/disable and replay is written to an audit record independent of the delivery telemetry.

How it is realised on Google Cloud. DynamoDB strongly consistent reads on the fan-out path. The authorisation service classifies actions into routine and elevated; elevated requires an assertion issued within a short window, and the API returns step_up_required so an automated client can respond correctly. Audit records are append-only with a 400-day TTL and are exported to the tenant's own SIEM.

What it buys

• An acknowledged unsubscribe is honoured, so the platform's worst regulatory failure has a structural answer.
• An account takeover cannot silently redirect a tenant's data with one routine-looking call.
• The audit trail survives the loss of the telemetry pipeline, because it is not in it.

What it costs

• A strongly consistent read on the fan-out path, which is a latency and cost line — affordable only because it is per event, not per attempt.
• The elevated-action list is a judgement made once and rarely revisited; drawn too narrowly, it protects nothing.

Choose differently when. Eventual consistency is fine where unsubscribe is not a compliance boundary and a few seconds of extra delivery is harmless. Uniform permissions are fine where every operator is already highly privileged.

Why it holds up over time. Both halves outlive their implementation: the rule that an acknowledged unsubscribe binds, and the rule that redirecting data needs more proof than filtering it. What could break the first is someone introducing a subscription cache to shave a few milliseconds off fan-out, which is why the consistency requirement is stated in view 06 rather than left in the datastore configuration.

Lesson. Grade permissions by what the action does to data, not by which screen it lives on. Two edits on the same form can have wildly different blast radii.

Every package used, in one table

The terms this package uses in a specific way, and what each one is doing in the architecture.