Architecture Views

Does the product's write path know anything about who is subscribed, or does it hand an event to a durable store and stop?

Context

The naive integration is the obvious one: when an order is placed, look up the subscriptions for `order.created`, and post to each. It works perfectly in development and it makes checkout latency a function of how many customers have subscribed, how many of them are slow today, and whether the subscription store is healthy. Every variant that keeps matching in the write path has the same defect in a smaller form — even a fast in-memory match makes a product transaction depend on a component whose whole job is to deal with the public internet. Meanwhile the product needs to acknowledge a payment in milliseconds and be right about it.

Decision

Intake validates, assigns an event id, persists the payload durably, and acknowledges — in that order — without consulting subscription state, endpoint health, or any external network. Subscription matching, entitlement evaluation and fan-out happen after the acknowledgement, on the platform's own time. An acknowledged event will be delivered or dead-lettered; it will never be lost. Producers hold events in their own transactional outbox, so an intake outage degrades the product briefly and visibly rather than silently dropping events.

How it is realised on Google Cloud

Intake is a stateless Fargate service behind an ALB across three AZs. The payload goes to S3 under an event id and the index row to DynamoDB in the same request path; the 202 is returned once both are durable. Fan-out consumes the committed-event stream asynchronously. The intake service holds no reference to the subscription table at all — the dependency does not exist rather than being unused.

Options weighed

• ChosenPersist, acknowledge, fan out afterwards: Intake latency becomes independent of subscription count and of consumer behaviour. Costs a second hop before the first attempt, which is why the p99 time-to-first-attempt is a stated objective rather than an afterthought.
• RejectedMatch subscriptions inside the write path: Makes a product transaction depend on subscription state and, at p99 fan-out of 40, on forty enqueue operations. The latency is a function of a customer's configuration.
• RejectedSynchronous delivery for a small set of 'critical' webhooks: The request that always comes, and the one that undoes the architecture. A synchronous webhook is a product transaction blocked on a stranger's server.
• Right elsewhereNo intake tier: producers write to the event log directly: Right where every producer is a trusted internal service on the same platform team. Here the size cap, the entitlement check and the acknowledgement contract need somewhere to live.

Consequences

What it buys

• Intake latency is independent of subscription count, so a tenant adding 500 endpoints cannot slow down checkout.
• An intake outage degrades the product visibly and briefly, rather than quietly losing events.
• The delivery plane can be scaled, deployed and broken independently of anything the product does.

What it costs

• One extra hop before the first attempt, which is why time-to-first-attempt carries its own p99 objective.
• The platform now owns durability for events it has acknowledged, which is what forces the RPO 0 requirement and the cross-region replication.

Choose differently when

Match in the write path when there is exactly one consumer, it is internal, and the write is already slow enough that one more call does not matter. A batch export to a single partner is in that category; a public webhook product never is.

Why it holds up over time

This is the decision least tied to any technology on this page and most likely to be eroded by a future feature request. Nothing about it depends on SQS, Fargate or AWS. What would break it is a 'synchronous webhook' option sold to one large customer, which is why the boundary is drawn explicitly in view 11 rather than implied.

When one consumer misbehaves, what is the smallest thing that is allowed to be affected?

Context

Multi-tenant platforms reflexively isolate by tenant, and here that is not enough. A tenant with five endpoints — production, staging, an analytics sink, a partner relay and one somebody forgot — will have one of them broken at any given time, and the other four must keep delivering. The failure being designed against is specific: an endpoint that accepts the connection and responds in fifteen seconds without ever erroring. Shared workers fill with in-flight attempts against that one endpoint, and the platform's availability becomes a function of the least reliable server in its customer base. A per-endpoint concurrency cap mitigates this and does not fix it, because a cap on in-flight attempts does not stop that endpoint's messages occupying the head of a shared queue while its circuit is open for six hours.

Decision

The endpoint is the unit of registration, configuration, queueing, concurrency, circuit breaking, health, backlog, accounting and failure. Two endpoints belonging to the same tenant are as isolated from each other as two endpoints belonging to different tenants. A worker leasing work for one endpoint can never be blocked by another endpoint's backlog. Per-tenant controls exist — a fan-out budget, an endpoint quota — but they are budgets, not the isolation boundary.

How it is realised on Google Cloud

Each endpoint has its own SQS FIFO message group (and, for the hot tail, its own queue — see ADR-03), its own concurrency cap enforced by the worker leasing logic, its own circuit state, its own dead-letter partition and its own attempt-log partition key. The delivery history API and every alert are scoped to an endpoint id.

Options weighed

• ChosenThe endpoint: Matches the failure that actually occurs. Costs cardinality: 40,000 of everything, which is what makes ADR-03 a real question.
• RejectedThe tenant: Cheaper and wrong in the common case. A customer's broken staging endpoint would degrade their own production deliveries, which is indefensible and impossible to explain.
• RejectedThe event type: Isolates a poison payload class but does nothing about a slow consumer, which is the dominant failure.
• RejectedA shared pool with per-endpoint concurrency caps only: The tempting middle. Caps bound in-flight attempts but not queue occupancy, so a large backlog behind an open circuit still delays everyone sharing the queue.

Consequences

What it buys

• A slow consumer's blast radius is exactly one endpoint, which is the only defensible answer to give a customer.
• Every operational number — backlog, success rate, retry share, cost — has a natural owner.
• Circuit breaking becomes safe to be aggressive about, because opening a circuit harms nobody else.

What it costs

• Cardinality of 40,000 across queues, circuits, health records and partitions, which is a real infrastructure constraint rather than a bookkeeping one.
• Fair scheduling across endpoints becomes a requirement, since arrival order no longer produces a sensible allocation.

Choose differently when

Isolate by tenant when tenants have one endpoint each, or when the consumer population is small and operationally competent — an internal event bus between a dozen teams does not need 40,000 of anything.

Why it holds up over time

The claim survives every plausible change of implementation. ADR-03 will be revisited when SQS limits, per-queue cost or the traffic distribution change; this record states the property that rewrite must preserve, which is why the two are separate decisions rather than one.

40,000 endpoints, each needing its own isolated stream — how many physical queues is that, and what breaks first?

Context

ADR-02 demands per-endpoint isolation without saying how it is realised, and the two obvious realisations fail differently. A queue per endpoint gives perfect isolation and a trivial mental model, and 40,000 queues is a management-plane problem, a per-queue cost line, and in most clouds a hard service limit that requires a quota conversation. One shared queue with a message group key per endpoint gives ordering isolation for free and throughput isolation not at all: a hot endpoint's messages still occupy the shared queue's consumers. The traffic distribution is the deciding fact and it is not known before launch — a webhook population is invariably long-tailed, with a handful of endpoints carrying most of the volume.

Decision

The mapping from endpoints to physical queues is an explicit, versioned scheme rather than an emergent property. The MVP uses a bounded set of shared SQS FIFO queues with the endpoint id as the message group key, giving ordering isolation and per-endpoint lease fairness, plus per-endpoint concurrency caps. Phase 2 adds automatic promotion: an endpoint whose volume or backlog crosses a threshold is moved to a dedicated queue, and demoted when it falls back. The promotion control loop is accepted as work rather than avoided.

How it is realised on Google Cloud

SQS FIFO queues with `MessageGroupId = endpoint_id`. Workers lease across groups with a fairness policy rather than draining a single group, so a large backlog in one group does not monopolise a worker. Promotion writes a queue assignment into the endpoint record; the planner reads it at enqueue time, and a drain window ensures no in-flight message is orphaned by a move.

Options weighed

• ChosenShared FIFO queues keyed by endpoint, promoting hot endpoints later: Ships without a quota negotiation, and buys time to measure the real distribution before committing to the expensive shape.
• DeferredOne queue per endpoint from day one: The cleanest isolation and the honest end state for the hot tail. Deferred because 40,000 queues is a service-limit and cost decision that should be made against measured traffic, not a guess.
• RejectedA single queue with in-worker fairness: Head-of-line blocking returns the first time an endpoint's backlog exceeds the prefetch window, which is the exact failure ADR-02 exists to prevent.
• RejectedA queue per tenant: Reintroduces same-tenant interference, which ADR-02 already refused.

Consequences

What it buys

• The MVP ships without a 40,000-queue quota conversation, and the promotion scheme has measured traffic to be designed against.
• Ordering isolation is available immediately, since the group key is the natural unit.

What it costs

• Throughput isolation in the MVP is bounded by lease fairness rather than by physical separation, which is a weaker promise than ADR-02 makes and is stated as such.
• Phase 2 adds a promotion/demotion control loop, including the drain window that stops a move orphaning in-flight work.

Choose differently when

Go queue-per-endpoint immediately when the endpoint population is in the low thousands, or when the queueing technology makes a queue nearly free. Stay with shared groups permanently when the traffic distribution turns out to be flat, which would be surprising and worth knowing.

Why it holds up over time

This is the decision on the page most likely to be replaced, and that is by design: it is the implementation of ADR-02 rather than a principle of its own. A future team should feel free to rewrite it entirely, provided the property in ADR-02 survives the rewrite.

What exactly is promised to a consumer about how many times they will see an event, and who is responsible for the difference?

Context

There is no way to deliver an HTTP request exactly once. The consumer can process an event and have the response lost on the way back; the platform then cannot distinguish that from a consumer that never received it, and must choose between retrying (duplicate) and not retrying (loss). Every webhook platform makes this choice, and the ones that describe their behaviour vaguely end up with consumers who did not implement deduplication because nobody told them clearly that they had to. The failure is silent and it happens in the consumer's ledger.

Decision

At-least-once is the published guarantee, stated plainly rather than implied. Every delivery carries an idempotency key that is stable across every retry of that delivery and across every replay of it, and the consumer documentation requires deduplication on that key. The platform prefers a duplicate to a loss in every ambiguous case and says so. A 2xx response that the consumer did not actually process is outside the platform's power to detect and is named as such rather than glossed over.

How it is realised on Google Cloud

The key is minted at fan-out and stored on the delivery row, sent as a header on every attempt, and reused verbatim by the replay service. The documented contract names the header, states the retry schedule, and says in one sentence that a consumer must be safe to receive the same key twice.

Options weighed

• ChosenAt-least-once with a stable key, published: Honest, implementable by the consumer, and makes replay safe by construction. Costs the platform nothing and the consumer a table.
• RejectedClaim exactly-once: Not deliverable over HTTP to a server the platform does not control. Its only real effect is that consumers skip deduplication.
• RejectedAt-most-once — never retry: Turns every transient consumer failure into permanent data loss, which for payment and order events is not a trade anyone would accept.
• Right elsewhereThin events with a callback, so duplicates are harmless: Right where payloads are large or sensitive, since the callback returns current state. Rejected here because it converts a delivery burst into an API burst against the same product, and current state is the wrong answer for anything auditable.

Consequences

What it buys

• Replay becomes safe without a new mechanism: it reuses the key the consumer is already deduplicating on.
• Regional failover can produce duplicates without that being an incident, because duplicates are inside the contract.
• The platform can be aggressive about retrying, which is what makes the 72-hour window valuable.

What it costs

• Every consumer must implement deduplication, and some will not. The mitigation is documentation and examples, not a stronger guarantee.
• The unclosable gap — consumer processed it, response lost — remains, and shows in the attempt log as a retryable failure that was actually a success.

Choose differently when

Move to at-most-once only where duplicate processing is catastrophic and loss is not, which is rare and usually indicates the event should not have been a webhook.

Why it holds up over time

The idempotency key is the field on this platform that can never change. Consumers write deduplication against it, and a change to its stability silently double-processes their transactions years later. Everything else here can be tightened; this cannot.

Are events delivered to an endpoint in order, and if so, in order of what?

Context

Consumers ask for strict per-endpoint ordering because it is obviously easier to reason about. It is also what makes one stuck delivery block everything behind it: a payload the consumer can never accept, sitting at the head of an ordered stream, is a 72-hour total outage for that endpoint rather than one dead letter. Per-resource ordering — all events about one order, one subscription, one pull request — confines the blocking to a single entity, at the cost of multiplying the number of ordered groups by orders of magnitude, well past what a queueing layer will address. No ordering at all is cheapest and pushes reconciliation onto the consumer, who will not do it and will then report a bug.

Decision

Ordering is a declared property of a subscription, with three modes and an explicit consequence attached to each: none (the default, events may arrive out of order and carry a timestamp for reconciliation), per-endpoint (strict, with head-of-line blocking stated at subscription time), and per-resource (Phase 2, ordered within a resource id). 'No ordering guarantee' is a legitimate, documented choice rather than an admission.

How it is realised on Google Cloud

The mode lives on the endpoint record. None and per-endpoint both use the endpoint id as the SQS FIFO message group key, differing in whether a blocked message halts the group or is moved aside; per-resource uses a composite group key and requires the queue mapping of ADR-03 to be revisited first.

Options weighed

• ChosenDeclared per subscription, default none: Puts the trade in front of the person who bears it, at the moment they can still choose. Costs a documented decision the customer has to make.
• RejectedStrict per-endpoint ordering for everyone: One poison payload becomes a total outage for that customer, and the platform is blamed for honouring exactly what was asked for.
• RejectedNo ordering, no option: Defensible and unsellable. Some consumers genuinely cannot reconcile, and telling them so is a lost integration.
• DeferredPer-resource ordering only: The best answer on the merits, and it needs an ordered-group cardinality the queueing layer cannot yet address. Phase 2, after ADR-03 is settled against real traffic.

Consequences

What it buys

• The customer who wants strict ordering can have it, knowing what it costs them.
• The default does not impose a 72-hour failure mode on customers who never asked for ordering.
• The timestamp on every event makes reconciliation possible for the default mode rather than theoretical.

What it costs

• Three delivery behaviours to implement, test and support instead of one.
• A customer on strict ordering who hits a poison payload has a bad day, and the platform's answer is a dead letter and an explanation rather than a fix.

Choose differently when

Offer only strict ordering where the event stream is a state-machine log the consumer replays — a change-data-capture feed, for instance — and only none where every event is independent and self-describing.

Why it holds up over time

The three modes will outlast their implementation. What could break this is a future team making per-endpoint ordering the default because it is what a vocal customer asked for, without carrying the head-of-line consequence into the default.

How does a consumer know a request came from this platform, and what happens to that assurance if the platform's own store is breached?

Context

A shared HMAC secret is what every consumer already knows how to verify, needs no key distribution beyond showing the secret once, and means the platform and the consumer both hold material that can forge a delivery — so a breach of the endpoint store is a forgery capability against every customer at once. An asymmetric signature over a published key removes the secret from the consumer entirely and makes rotation a publish rather than a coordinated change, at the cost of asking every consumer to fetch, cache and verify against a key, which is more code and one more thing to get wrong. The deciding fact is that the population includes people integrating in an afternoon, and a signature they cannot verify is a signature they will skip.

Decision

HMAC-SHA-256 over a canonical string of timestamp plus the exact raw request body, under a per-endpoint secret, with the canonical string published as a contract rather than described in prose, and worked verification examples in the languages the customer base actually uses. The timestamp is inside the signed material so a captured request cannot be replayed indefinitely, and a recommended acceptance window is published. Asymmetric signatures are Phase 3, offered alongside rather than replacing HMAC.

How it is realised on Google Cloud

The per-endpoint secret is generated as a KMS data key, stored wrapped, and shown to the customer exactly once. Workers decrypt with a short-lived cache and sign each attempt over the raw bytes being sent. Both active secrets produce a signature during a rotation overlap (ADR-07). The canonical string is frozen by a contract test in CI, because changing it silently breaks every consumer's verification at once.

Options weighed

• ChosenHMAC over a published canonical string: Universally implementable, no key distribution, no fetch. Costs the platform holding forgery-capable material for every customer.
• DeferredAsymmetric signature over a published key: Strictly better against a platform-side compromise and makes rotation unilateral. Phase 3, offered in addition — the verification burden on a consumer integrating in an afternoon is the blocker, not the cryptography.
• RejectedmTLS to the consumer's endpoint: Strong, and asks every consumer to run a TLS terminator that can present and validate client certificates. A large fraction of the population cannot.
• Right elsewhereNo signature; rely on a secret path component: Adequate for a low-value internal integration where the URL itself is the secret. Not adequate for anything that moves money or grants access.

Consequences

What it buys

• Verification is a handful of lines in any language, which is what makes it actually happen.
• Rotation is possible without contacting the consumer, because two secrets are live at once.
• The canonical string being a frozen contract means a refactor of the request builder cannot silently break 40,000 integrations.

What it costs

• A breach of the endpoint store is a forgery capability against every customer, mitigated only by wrapping the secrets under KMS and by the Phase 3 asymmetric option.
• The platform cannot contain a secret compromise without the customer's cooperation, since only they can stop trusting the old signature.

Choose differently when

Go asymmetric first when the consumer population is sophisticated, when the platform's own compromise must be containable unilaterally, or when a regulator requires non-repudiation the platform itself cannot produce.

Why it holds up over time

The canonical string, not the algorithm, is the thing that must survive. SHA-256 will be replaced; the rule that the signature covers the timestamp and the exact raw bytes, and that the string is published rather than described, should not be. Signing a re-serialised body is the defect that would quietly break everything, which is why view 13 shows the verifier recomputing over raw bytes.

How does a signing secret change without a coordinated cut-over between the platform and a consumer who may be asleep?

Context

A single active secret makes rotation an atomic, coordinated event: the platform switches, and every request between that moment and the consumer's own deploy fails verification. For a consumer who rotates during their own business hours in a different timezone, that window is hours of dead letters. The failure is also the worst kind, because it looks exactly like an attack: valid-looking requests failing signature verification.

Decision

Each endpoint may have two active secrets, bounded by `active_from` and `active_until`. During an overlap the platform sends a signature for each, and the consumer accepts a request if either verifies. The customer rolls a secret whenever they like, migrates their code at their own pace within a seven-day overlap, and retires the old one when they are ready. A secret is never displayed again after creation.

How it is realised on Google Cloud

Both secrets live on the endpoint record as KMS-wrapped data keys with validity windows. The worker reads both, signs twice, and sends both signature values in one header. Rotation is a data change, effective within the propagation bound, not an operational procedure requiring coordination.

Options weighed

• ChosenTwo active secrets with a bounded overlap: Rotation becomes a customer-paced change with no failed deliveries. Costs a second signature computation per attempt and a slightly larger header.
• RejectedSingle secret, atomic switch: Every rotation is an outage window whose length is the consumer's deploy cadence, and it presents as a signature attack.
• RejectedUnbounded number of active secrets: A secret that never expires is a secret nobody retires. The bounded window is what makes rotation finish.
• Right elsewhereKey identifier in the header, consumer looks up which secret: The right shape once signatures are asymmetric and keys are published, which is Phase 3. It asks a consumer to maintain a key store, which the HMAC population will not.

Consequences

What it buys

• A leaked secret can be rotated immediately without the platform having to coordinate anything with the customer.
• Rotation produces no failed deliveries, so customers actually do it.
• The overlap window is a data property that can be shortened per endpoint during an incident.

What it costs

• Two HMAC computations per attempt at 45,000 attempts/s, which is measurable but small relative to the network cost.
• A straggler who never migrates keeps a compromised secret valid until `active_until`, which is why the window is bounded rather than open-ended.

Choose differently when

A single secret with an atomic switch is fine where the platform and the consumer are the same organisation and can deploy together. Beyond that boundary, overlap is not optional.

Why it holds up over time

The overlap mechanism survives a change of algorithm and a change to asymmetric keys, where it becomes two published keys rather than two secrets. The seven days is a parameter; the two-live-credentials shape is the decision.

Who decides how often a failing endpoint is retried, and what happens the moment it comes back?

Context

Per-subscription retry configuration is genuinely useful: a consumer who knows their maintenance window is an hour wants different behaviour from one running a serverless handler that cold-starts. It also lets a customer configure a schedule that is a self-inflicted denial of service, makes the platform's retry capacity unpredictable at 45,000 attempts per second, and turns the support answer to 'when will you retry?' into 'it depends on what you set'. The second half of the problem is the opposite: when a large consumer returns after four hours down, draining their backlog at full rate is the platform causing that customer's second outage.

Decision

One published schedule: twelve attempts over 72 hours, full-jittered exponential backoff from a five-second base, capped at six hours between attempts. `Retry-After` on 429 and 503 is honoured up to a one-hour ceiling, because a consumer telling the platform when to come back is the most useful signal it will ever get. A circuit opens after twenty consecutive failures and probes once every five minutes with a single attempt. On recovery, resumption ramps from 10% to 100% of the endpoint's concurrency cap over five minutes. Named retry profiles, rather than free configuration, are Phase 2.

How it is realised on Google Cloud

Backoff is computed per delivery and written to `next_attempt_at`; SQS visibility timeout carries the short intervals and a scheduler re-enqueues the long ones. Full jitter rather than fixed exponential specifically to avoid a synchronised retry wave when a popular consumer returns. The ramp is enforced by the worker's per-endpoint concurrency governor, not by the queue.

Options weighed

• ChosenOne published schedule, with Retry-After honoured: Predictable for capacity, documentable once, and the one consumer-supplied signal that is actually authoritative is still respected.
• RejectedFree per-subscription configuration: Unplannable capacity, an unanswerable support question, and a customer-configurable way to hurt themselves.
• DeferredNamed profiles (aggressive / standard / patient): Phase 2. Captures most of the real variance with a bounded set of behaviours the platform can still plan for.
• RejectedFixed-interval retry: Simple, and it synchronises: every consumer that came back at the same moment retries at the same moment. Jitter is the whole point.

Consequences

What it buys

• Retry capacity is plannable, and 'when will you retry?' has a one-sentence answer.
• Full jitter means a popular consumer's recovery does not produce a thundering herd from the platform.
• The ramp means the platform is never the cause of a consumer's second outage.

What it costs

• A consumer whose operational reality does not fit the schedule has no recourse until Phase 2.
• 72 hours of retries for a permanently dead endpoint is capacity spent producing nothing, which is what the auto-disable in ADR-14 bounds.

Choose differently when

Allow free configuration where consumers are internal, few, and accountable for the capacity they consume. On a public platform with 40,000 endpoints, the schedule is infrastructure.

Why it holds up over time

The numbers will be argued about forever and should be. What must not drift is that the schedule is published, uniform and the platform's — the moment it becomes per-tenant configuration, capacity planning and support both stop being sayable in one sentence.

When a delivery finally fails for good, who finds out, and when?

Context

The worst outcome a webhook platform can produce is an integration that stops working and nobody notices — discovered a week later by a customer's finance team reconciling numbers that do not add up. The temptation is to notify on the state change that feels most significant, which is the endpoint being disabled after 72 hours. By then the customer has lost three days of events and the platform's message arrives as an accusation rather than a warning. The first dead letter is the actionable moment; the hundredth is noise.

Decision

The tenant is notified when deliveries begin dead-lettering, not when the endpoint is eventually disabled. A permanent failure — a 4xx that is not 408 or 429 — is never retried and never silent: its owner is told, because a misconfigured path or a revoked credential produces permanent failures indefinitely and produces them quietly. Notification goes through a channel that does not depend on the customer's endpoint being healthy. Subsequent dead letters within a window are aggregated rather than each producing a message.

How it is realised on Google Cloud

The first dead letter for an endpoint within a rolling window raises an event to the product's existing notification service, which reaches the same people who receive every other account notification. The endpoint's health state moves to degraded or failing and becomes visible in the console and in the delivery health API before any notification is sent, so a customer watching their own dashboard sees it first.

Options weighed

• ChosenNotify at the first dead letter, aggregate the rest: The earliest moment the customer can act. Costs some false alarms for endpoints that fail occasionally and recover.
• RejectedNotify only on auto-disable: Three days late. By then the message is a post-mortem rather than an alert.
• RejectedNotify on every dead letter: A broken endpoint produces thousands. The tenth message trains the customer to filter the channel, which loses the first message on the next incident.
• Right elsewhereNo notification; expose it in the API and let the customer alert: Right for a platform whose consumers are all sophisticated operators with monitoring of their own. This population includes people who integrated in one afternoon.

Consequences

What it buys

• The customer learns at the earliest actionable moment, on a channel that works when their endpoint does not.
• A permanent failure has an owner instead of accumulating silently.
• Notification volume is bounded, so the channel stays credible.

What it costs

• A dependency on a notification service outside the delivery boundary, accepted because it is not on the delivery path.
• Occasional false alarms from endpoints that dead-letter once and recover, which is the correct side to err on.

Choose differently when

Suppress notification entirely where every consumer has been contractually required to monitor delivery health themselves, and where the platform has a way to verify they do.

Why it holds up over time

Independent of every technology here. The failure it prevents — silent, discovered late, discovered by someone else — is the same failure in any asynchronous integration, and the instinct to notify on the dramatic state change rather than the early one is equally durable.

How does a customer get back the events they missed, without that recovery being its own outage?

Context

A consumer down for four days has a backlog waiting for them. Handing it over is the point of retaining it, and handing it over at full speed does to their system exactly what the original outage did — with the added difficulty that they asked for it, so it looks like their fault and feels like the platform's. There is also a request that always arrives and sounds unreasonable until you think about it: 'we processed those events and then lost our database, can you send them again?' That is not a failure of this platform, and refusing it is refusing the only copy of data the customer has.

Decision

A replay is a new delivery carrying the original event id and the original idempotency key, so the consumer's existing deduplication still protects them. Replay of successfully delivered events within the retention window is supported, not only of dead letters. Replay is rate-capped per endpoint, requires explicit confirmation above a declared size, and runs on a separate worker pool so it cannot starve first-attempt delivery. A dead letter is never deleted before its retention expires, including when the endpoint is deleted, unless the tenant explicitly purges it.

How it is realised on Google Cloud

The replay service reads the dead-letter index or the delivery records, re-enqueues to the endpoint's message group with the original key, and writes `replayed_as` on the original so the relationship is traceable. Replay workers are a distinct Fargate service with their own concurrency budget. The bulk-replay confirmation is a product surface, not a rate limit — the customer is being told what they are about to do.

Options weighed

• ChosenNew delivery, original key, rate-capped, separate pool: Safe by construction because it reuses ADR-04's contract rather than inventing a second one. Costs a separate worker pool and a confirmation dialog.
• RejectedReplay with a fresh idempotency key: Would double-process every replayed event in a correctly built consumer. The key being stable is the entire point.
• RejectedReplay only dead letters: Refuses the 'we lost our database' request, which is the case where the platform holds the only remaining copy.
• RejectedReplay at full rate, let the consumer rate-limit with 429: Relies on a consumer having a working rate limiter at exactly the moment they are least likely to. The platform knows better and should act on it.

Consequences

What it buys

• Self-service recovery with no support ticket and no new safety mechanism.
• A bulk replay is survivable for the consumer, and traceable afterwards through `replayed_as`.
• Replay cannot degrade first-attempt delivery for anyone, including the tenant replaying.

What it costs

• A separate worker pool that is idle most of the time.
• The retention window becomes a hard customer-facing contract, because it silently defines what can be replayed at all.

Choose differently when

Replay at full rate where the consumer is internal, their capacity is known, and the platform can coordinate the drain with them directly.

Why it holds up over time

Replay's safety comes entirely from ADR-04's stable key. As long as that holds, replay can be reimplemented freely. What would break it is a future 'replay with a fresh id' option added to work around a consumer who deduplicates incorrectly, which would fix one customer and silently endanger every other.

Where do deliveries appear to come from, and what does it take to change that?

Context

A meaningful share of enterprise consumers will only accept inbound traffic from allowlisted source addresses. Publishing the range is therefore a product requirement, not an operational convenience. The consequence is usually discovered too late: once published, the range lives in tens of thousands of firewall rules the platform cannot see, cannot query and cannot update. An infrastructure ticket that moves a NAT gateway then presents, to the customer, as the platform being unreachable — and to the platform, as thousands of endpoints simultaneously timing out, which is indistinguishable from a widespread consumer problem.

Decision

The egress range is a versioned, published, documented interface, changed only through a review gate in the release pipeline with a published deprecation window and dual-range operation during it. Both the active and the standby region's ranges are published before launch, so a regional failover is not also an allowlist change for 40,000 consumers. The range is deliberately narrow and over-provisioned so that growth does not force a change.

How it is realised on Google Cloud

Dedicated NAT gateways on a reserved /28 per region, with the range asserted in Terraform and a CI gate that fails a plan changing it without an accompanying interface-change record. The ranges are served from a machine-readable endpoint so consumers can automate their allowlists.

Options weighed

• ChosenNarrow published range per region, both published upfront, changed via review: Makes failover usable and makes an accidental change impossible. Costs reserved address space that may sit unused for years.
• RejectedPublish nothing; deliver from whatever address the cloud assigns: Loses every enterprise consumer with an allowlist requirement, and there is no way to win them back later without the range.
• DeferredPer-tenant egress addresses: What the most demanding security reviews ask for, and a multiple of the NAT footprint. Phase 3, for the tenants who will pay for it.
• Right elsewhereSigned requests only, no address guarantee: Cryptographically sufficient and organisationally insufficient. A consumer's network team allowlists addresses; they do not read the signature documentation.

Consequences

What it buys

• Enterprise consumers can integrate at all.
• A regional failover is a routing change rather than a coordinated allowlist migration across the customer base.
• Delivery traffic is attributable from outside the platform, which helps when a consumer reports unexpected requests.

What it costs

• Reserved address space in two regions, mostly unused.
• Any future need to change the range is genuinely expensive, which is the cost of having made the promise.

Choose differently when

Skip the published range entirely where every consumer is a cloud-hosted service with no ingress allowlist, which is a bet on the customer base that gets harder to unwind every year.

Why it holds up over time

This is the hardest thing on the platform to change and the decision a future team will be most grateful for. It survives a change of cloud only if the replacement can present the same addresses, which is itself an argument for keeping the range narrow.

A worker fleet connects to URLs strangers chose. What stops one of those URLs being something inside the estate?

Context

This is the platform's defining security property, and it is unusual: the service is, by design, a machine that makes HTTP requests to arbitrary addresses supplied by unauthenticated third parties. Every classic server-side request forgery technique applies directly — a URL resolving to a private range, DNS rebinding between validation and connection, a redirect to the instance metadata service, a scheme downgrade. An address-validation function handles all of them correctly until the day it has a bug, and the population of ways to express an address is large enough that it eventually will.

Decision

Address validation is implemented thoroughly — resolution checked at registration and immediately before every connect, re-checked after every redirect, redirects capped at three, scheme downgrade refused, signature headers re-signed rather than forwarded to a redirected host — and it is explicitly not the containment. Delivery workers run in egress-only subnets with no route to the capture plane, the control plane, the product VPCs or the instance metadata service. If the guard has a defect, the packet has nowhere to go.

How it is realised on Google Cloud

A dedicated VPC with delivery subnets whose route table contains a default route to the NAT gateway and nothing else; no peering, no transit attachment, no VPC endpoints to internal services, metadata service disabled at the task level. The address guard suite runs on every build as a release gate (view 16), covering rebinding, redirect chains and private-range expression forms.

Options weighed

• ChosenValidation plus network isolation, isolation as the containment: Two independent controls with different failure modes. Costs a separate network and the operational friction of a fleet that cannot reach internal services.
• RejectedValidation only: One control, one bug away from an internal request forwarder with the platform's own credentials.
• DeferredEgress through a proxy fleet that enforces policy: Centralises the guard and adds a hop on every attempt at 45,000/s. Worth revisiting if per-tenant egress (ADR-11) is built, since the proxy becomes the natural place for it.
• Right elsewhereAllowlist consumer addresses at registration: Correct for a small partner integration with a handful of known counterparties. Unworkable for a self-service product where the point is that anyone can register a URL.

Consequences

What it buys

• A defect in the address guard is contained by routing rather than exploited.
• The security review has a structural answer rather than a code-review answer, which is what makes it repeatable.
• Workers hold no credentials that would be useful inside the estate, because they have nowhere to use them.

What it costs

• A separate network to operate, and workers that cannot reach internal services for convenience — every dependency they need must be deliberately provided.
• Debugging from a delivery worker is harder, by design.

Choose differently when

Rely on validation alone only where the workload is not attacker-directed — a crawler over a curated list, for instance. The moment a stranger supplies the address, the network has to be the answer.

Why it holds up over time

The reasoning outlives AWS: whatever the platform runs on, the component that connects to attacker-chosen addresses should be somewhere that a successful redirection reaches nothing. What would erode it is a future feature giving the delivery worker a legitimate reason to call an internal service — at which point the right answer is to move that work out of the worker, not to open the route.

When a developer says 'we're not receiving your webhooks', how do they find out why without opening a ticket?

Context

Every integration platform's support load is dominated by one question, and the honest answer is usually 'your server returned a 403 four thousand times'. Without a delivery history the customer can read themselves, answering it requires an engineer to query production, which does not scale past a few hundred integrations. The mirror-image problem is a customer reporting that deliveries are slow, when the latency is entirely their own handler — with one aggregate number, the platform has no way to say so and no way to prove it.

Decision

Every attempt is recorded with its timestamp, duration, response status, error class and a bounded prefix of the response body, and that history is exposed to the endpoint's owner for the retention window, including the exact request that was sent so a developer can replay it locally. Platform time and consumer time are measured and reported as separate quantities at every stage. A per-endpoint dashboard and an equivalent API expose delivery rate, success rate, latency and backlog depth, so a customer can alert on their own integration health rather than discovering a break from their users.

How it is realised on Google Cloud

Attempt records in DynamoDB partitioned by endpoint with a 90-day TTL; the request inspector reconstructs the sent bytes from the stored payload and the recorded headers. Fields the product classifies as sensitive are redacted in the history while the signature remains valid over the bytes actually sent. Spans separate queue time, platform processing and time-on-the-wire.

Options weighed

• ChosenFull per-attempt history, exposed to the customer, with the latency split: Turns the dominant support question into a self-service one. Costs the storage of 45,000 attempt records per second and a redaction path.
• RejectedSummary counters only: Cheap, and it answers 'is it failing' without answering 'why', which is the only question anyone asks.
• RejectedHistory retained internally, surfaced only via support: Scales to roughly the number of engineers willing to run production queries.
• DeferredStream attempt records to the customer's own log platform: Valuable for sophisticated consumers and a poor primary interface, because it requires the customer to build something before they can see anything.

Consequences

What it buys

• A developer diagnoses their own integration, which is what makes 40,000 endpoints supportable.
• 'Is it you or us' has a measured answer rather than an argument.
• The request inspector shortens the signature-verification trough in view 04, which is where new integrations actually stall.

What it costs

• Attempt history is the platform's largest write volume and a significant storage line.
• Redaction must be correct, since the history displays data the endpoint's owner may not be entitled to see in full.

Choose differently when

Summary counters are enough where consumers are internal teams with access to the same observability platform, and where a shared dashboard already answers the question.

Why it holds up over time

The latency split is the measurement that ages best: scale changes what the platform looks like, not what it is judged on. A future team with different storage and ten times the traffic will still need to answer whether a slow integration is theirs.

What happens to an endpoint that has been failing for three days — and who decides?

Context

An endpoint that has failed continuously for 72 hours is almost always abandoned, and retrying it forever is capacity spent producing nothing. Disabling it is also the platform unilaterally breaking a customer's integration, potentially in the middle of that customer's own extended outage, at a moment when nobody is reading email. Both the action and the inaction have a real cost, and there is no option without one. The secondary question is worse: on re-enable, delivering three days of backlog into a system that has just been repaired can repeat the original failure, and skipping it silently loses data the customer believed was safe.

Decision

An endpoint is auto-disabled after 72 hours of continuous failure with zero successes. Its undelivered backlog is retained for the dead-letter window regardless. The tenant is notified at the first dead letter — up to 72 hours before the disable — on a channel that does not depend on the endpoint. Re-enable is an explicit human action with an explicit choice: deliver the backlog, skip it, or resume from a chosen point. The platform does not make that choice on the customer's behalf.

How it is realised on Google Cloud

Health state transitions are computed from recent attempts (view 18). The disable writes an audit record and a notification; the backlog stays in the dead-letter store under its own retention. Re-enable presents the backlog size and the estimated drain time before the choice is made, because a customer choosing to deliver 400,000 events should be told that first.

Options weighed

• ChosenDisable at 72 h, retain the backlog, explicit re-enable choice: Bounds the wasted capacity without losing the data, and puts the risky decision with the person who understands their own system.
• RejectedNever disable: An abandoned endpoint accrues retries indefinitely, and the population of abandoned endpoints only grows.
• RejectedDisable and discard the backlog: Cheapest, and it converts the platform's operational convenience into the customer's data loss.
• RejectedPlatform chooses on re-enable (always deliver the backlog): Sounds generous and is the mechanism by which a just-repaired system is broken again.

Consequences

What it buys

• Wasted retry capacity is bounded.
• No data is lost by the disable itself, so the decision is reversible for 30 days.
• The customer makes the dangerous choice, informed, at a moment of their choosing.

What it costs

• A customer whose outage exceeds 72 hours has their integration disabled during it, which is a bad experience however well it is handled.
• The whole justification rests on the notification being reliable, which is an assumption until measured.

Choose differently when

Never disable where endpoints are few, owned internally, and the retry cost is negligible. Disable much sooner where the consumer population is untrusted and retry capacity is the binding constraint.

Why it holds up over time

The shape — bound the cost, keep the data, hand the risky decision to whoever understands the consequences — outlives every number in it. The 72 hours is a parameter and should move with evidence; the principle should not.

When a customer says an event never arrived, what does the platform consult, and what can it prove?

Context

The convenient design keeps one store, where a delivery row is mutated as attempts proceed and holds the latest outcome. It answers 'what is the state now' well and everything else badly. A customer dispute is a question about history — was it sent, when, how many times, what did our server say — and a mutated row has thrown that away. Deriving one record from the other is also not possible in either direction: the accepted event does not know how many attempts it took, and the attempt log does not contain the payload.

Decision

The accepted-event store is the system of record for what the platform was asked to send. The attempt log is the system of record for what it actually did. Neither is derived from the other, both are immutable once written, and the queue state is derived from the second — reconstructible after a total loss of the queueing layer, without operator input, without re-sending anything already recorded as delivered.

How it is realised on Google Cloud

Events are written once and never mutated; a correction is a new event. Attempt records are append-only, partitioned by endpoint, with a 90-day TTL for the console view and a 13-month aggregate. Delivery rows carry current state as a projection over the attempt log rather than as independent truth, so a lost delivery row is rebuildable.

Options weighed

• ChosenTwo immutable records, queue state derived from the attempt log: Disputes are answerable, and queue loss is a rebuild rather than an incident. Costs the write volume of an append-only log at 45,000/s.
• RejectedOne mutable delivery row: Cheap, and it throws away exactly the history a customer dispute is about.
• RejectedAttempt history in the observability platform rather than a store: Telemetry is sampled, dropped under load and retained on a different schedule. A record that can be sampled is not a record.
• Right elsewhereEvent log as the only store, deliveries recomputed: Elegant where consumers are internal and history is not disputed. Here the attempt history is a customer-facing product in its own right (ADR-13).

Consequences

What it buys

• A dispute is answered from evidence, with timestamps and response codes, rather than from recollection.
• Queue-layer loss is a rebuild, which is why the queues carry no recovery objective of their own.
• The customer-facing delivery history is a view over the record rather than a parallel copy that can disagree with it.

What it costs

• Append-only writes at attempt volume are the platform's largest write load and a major storage line.
• Two retention schedules to reason about, and they are deliberately different.

Choose differently when

A single mutable row is fine where history is never disputed and the queue is itself durable and authoritative — an internal job runner, for example.

Why it holds up over time

The separation survives any storage technology. What would erode it is a performance optimisation that starts updating the delivery row in place and stops writing an attempt record for a successful first attempt — which is the majority case and therefore exactly the record that would be missing when a customer asks about it.

How fresh must subscription state be when an event is fanned out, and who is allowed to change where a tenant's data goes?

Context

Two problems share one answer. The first is consistency: an unsubscribe the customer has been told took effect, followed by a delivery, is a data-protection incident rather than a scheduling lag — and continued delivery after a tenant deletion is the worst-consequence failure on this platform. A cached subscription projection is cheap and fast and makes exactly that mistake. The second is authority: changing an endpoint's URL redirects the tenant's data somewhere new, and in most permission models it is the same routine operation as changing which event types are sent. An account takeover would use it, and it would look like ordinary configuration.

Decision

Subscription state is read strongly consistent at fan-out time — once per event rather than once per attempt, which is what makes it affordable. When the subscription store is unavailable for reads, delivery continues on last known state and subscription writes are refused, except unsubscribes and deletions, which fail closed. Changing an endpoint URL requires a higher authority and a fresh authentication than changing its event-type selection, returned as a machine-readable step-up rather than a generic denial. Every subscription change, secret rotation, enable/disable and replay is written to an audit record independent of the delivery telemetry.

How it is realised on Google Cloud

DynamoDB strongly consistent reads on the fan-out path. The authorisation service classifies actions into routine and elevated; elevated requires an assertion issued within a short window, and the API returns `step_up_required` so an automated client can respond correctly. Audit records are append-only with a 400-day TTL and are exported to the tenant's own SIEM.

Options weighed

• ChosenStrong consistency at fan-out, step-up for URL changes, independent audit: Both failures are structural rather than procedural. Costs a consistent read per event and one more authentication step on a rare action.
• RejectedCached projection with a short TTL: Fast, and it delivers events to an endpoint the customer unsubscribed thirty seconds ago — which is precisely the case that matters.
• RejectedUniform permissions for all endpoint edits: Makes redirecting a tenant's entire event stream as easy as toggling an event type.
• Right elsewhereApproval workflow for every subscription change: Right in a regulated estate with a change board. On a self-service developer platform it would stop people integrating.

Consequences

What it buys

• An acknowledged unsubscribe is honoured, so the platform's worst regulatory failure has a structural answer.
• An account takeover cannot silently redirect a tenant's data with one routine-looking call.
• The audit trail survives the loss of the telemetry pipeline, because it is not in it.

What it costs

• A strongly consistent read on the fan-out path, which is a latency and cost line — affordable only because it is per event, not per attempt.
• The elevated-action list is a judgement made once and rarely revisited; drawn too narrowly, it protects nothing.

Choose differently when

Eventual consistency is fine where unsubscribe is not a compliance boundary and a few seconds of extra delivery is harmless. Uniform permissions are fine where every operator is already highly privileged.

Why it holds up over time

Both halves outlive their implementation: the rule that an acknowledged unsubscribe binds, and the rule that redirecting data needs more proof than filtering it. What could break the first is someone introducing a subscription cache to shave a few milliseconds off fan-out, which is why the consistency requirement is stated in view 06 rather than left in the datastore configuration.