Programmable Messaging API

Architecture Views

22 views, in reading order. Every view ships three ways: an HTML page, an SVG that re-opens in diagrams.net fully editable, and draw.io source.

Twenty-two views of the platform behind every login code and delivery notification: one HTTP call in, a message on a handset out, through carriers the platform does not own and cannot see inside. Read in order — the boundary first, then the people it happens to, then the structure that keeps an accepted message honest when the carrier goes quiet. Every number in the cards is a stated assumption; the requirement document names them as such.

Context and scope

What sits inside the boundary, who it serves, and which of its dependencies belong to somebody else.

People and journeys

The cast, and the three journeys that decide whether the architecture is right: a login code, a developer's first non-delivery, and the word STOP.
03 The person holding the handset Message recipient never signed up with us Goal — Get the code while the login page is still open, and have STOP actually stop it. Core journeys Receive a login code the six-digit moment Text STOP and be left alone Tenant builders and operators App developer 40,000 tenants Goal — Send my first message in ten minutes, and find out why one did not arrive without opening a ticket. Core journeys Send the first message Diagnose a non-delivery Campaign operator bulk traffic Goal — Get four million messages out inside my window, and know whether they are paced or stuck. Core journeys Run a paced campaign Read queue age by route Compliance officer 190 jurisdictions Goal — Prove what we sent to this number, when, and on whose recorded consent. Core journeys Answer a regulator Freeze a sender identity Carriers and the people who watch them Carrier or aggregator 220 routes Goal — Receive traffic inside the window we contracted, from a sender identity we registered. Core journeys Accept paced submissions Return delivery receipts Platform SRE on call Goal — Know within minutes that a route is accepting everything and delivering nothing. Core journeys Catch a silent route Drain a dead bind Who the Platform Is For, and What They Get To Do Person or role Journey / task External / third party The paying customer is the developer; the person the product happens to is the recipient. v 1.0 · owner Integration Platform Architecture Actors and Their Journeys Six actors, and the goal each of them would state in their own words. HTML page SVG draw.io
05 App developer first integration, no telecom background Goal — Send a message, then account for one that did not arrive Trigger — A product requirement to send order confirmations Done when — Shipped, and able to answer support without a ticket to us 1 · Read the docs 2 · First send 3 · It says accepted 4 · Customer says nothing came ◆ moment of truth 5 · Find out why ◆ moment of truth 6 · Ship What they do Picks a sender identity POSTs one message Logs the message id Searches by their own id Reads the attempt trail Adds the callback What the platform does Dry-run shows route and price Rejects unroutable at submit Accepted is a promise, not a delivery Lookup by client reference Route, attempts, receipts, reason Signed callback with DLQ How it feels Confident Uneasy Blaming us Where it hurts Sender rules differ by country Accepted reads as delivered No answer sounds like no system Carrier reason codes are jargon What the architecture owes it Rules are data, not support lore Fast, specific rejections Status vocabulary says accepted Unknown is a named answer Reason mapped to plain words Pull reconstructs what push missed Journey — Developer Sends One Message, Then Has To Explain One The trough is a customer complaint the developer cannot answer; everything in act 5 exists to move it. v 1.0 · owner Integration Platform Architecture Journey — First Send, First Complaint The developer's trough is a customer complaint they cannot answer. HTML page SVG draw.io

Structure

The layers, the deployable parts, and every interface the platform offers or consumes.
08 Google Cloud — messaging project Tenant-facing plane Global load balancer Cloud Armor Submit service Cloud Run Status service Cloud Run Callback dispatcher Cloud Run Admission and ownership Admission service Compliance evaluator Message ledger Cloud Spanner Body store Cloud Storage State projection Bigtable Delivery plane Scheduler class lanes Lane topics Pub/Sub Route pacing queues Cloud Tasks Attempt executor Cloud Run Lease and quality store Memorystore Session plane (stateful, pinned) SMPP session pods GKE StatefulSet Static egress Cloud NAT Carrier credential custody Secret Manager + KMS Ingest plane Receipt endpoint Inbound endpoint Receipt and inbound streams Pub/Sub Correlate and apply Quarantine Control plane and evidence Registry service routes, senders, rules Consent store Spanner Analytics and evidence BigQuery Audit archive Bucket Lock Carriers and aggregators 220 routes Channel business APIs WhatsApp, RCS Metering and billing HTTPS fails closed append then ack pace submit bind HTTPS receipts no match Container View — Services, Stores and the Boundaries Between Them Interface / broker Application we own Security / platform Data store Queue / topic Risk / gap External / third party synchronous event / async two-way failure / alternate Omitted: console, IaC and the per-service sidecars. The session plane is the only stateful workload. v 1.0 · owner Integration Platform Architecture Container View The deployable parts on Google Cloud, and the one plane that cannot be stateless. HTML page SVG draw.io

Data

What is authoritative, what is derived, what outlives what, and the model that makes an unknown outcome representable.
12 tenant tenant_id PK plan entitlement_mps residency_region credit_balance sender_identity sender_id PK tenant_id FK -> tenant type long|short|alpha|toll registrations jsonb throughput_ceiling route route_id PK channel jurisdiction provider price_per_segment window_mps consent_event consent_event_id PK tenant_id FK -> tenant destination_hash sender_id FK -> sender state granted|withdrawn evidence recorded_at message message_id PK tenant_id FK -> tenant client_reference UQ per tenant destination_hash sender_id FK -> sender channel_requested traffic_class ttl_expires_at segments_quoted accepted_at attempt attempt_id PK message_id FK -> message seq route_id FK -> route provider_reference submit_outcome segments_charged submitted_at message_body body_id PK message_id FK -> message object_uri kms_key purge_after receipt receipt_id PK attempt_id FK -> attempt NULLable provider_reference carrier_status carrier_reason carrier_timestamp received_at message_state message_id PK FK -> message state accepted..unknown terminal_at derived_from_attempt // projection, not authority inbound_message inbound_id PK route_id FK -> route provider_reference UQ per route sender_id FK -> sender keyword_actioned received_at rated_usage_event usage_event_id PK attempt_id FK -> attempt tenant_id FK -> tenant route_id FK -> route segments amount 1 : N at submit 1 : N 1 : N 1 : N 1 : N 1 : 1 1 : 0..1 1 : N 1 : 1 Data Model — Message, Attempt, Receipt, Consent receipt.attempt_id is nullable on purpose: an uncorrelatable receipt is kept, not dropped. v 1.0 · owner Integration Platform Architecture Data Model Message, attempt, receipt, consent — and the nullable foreign key that is the point of the view. HTML page SVG draw.io

Runtime

What actually happens on a send, on a receipt, per channel, and in the case nobody can resolve.

Operations

Where it runs, how change ships, what is watched, and how a carrier route earns and loses traffic.

Assurance

The trust boundaries, and the four separate things that have to prove themselves before a message leaves.

Architecture One-Pager

The problem, the shape of the answer, the decisions that carry it, and what a prototype should prove.

The accepted message is the unit of durability and the carrier attempt is disposable: delivery state is a monotonic projection over attempts and late-arriving receipts, never a field the submit path writes.

Everybody has used this system today without knowing its name. The six-digit code that arrives when you log into your bank, the "your driver is 2 minutes away" text, the prescription reminder, the airline's gate-change message. None were sent by a human, and almost none by the company whose name is on them. They went through a programmable messaging API: one HTTP endpoint a developer calls with a destination, a body and a sender identity, and a platform behind it that turns that call into a message on a handset anywhere in the world, through carriers it does not own, under regulations it does not write. The product surface is one POST. The architecture is hard for exactly one reason, and it is not scale: the platform cannot see whether the message arrived. It hands the message to a carrier or an aggregator, and what comes back is a receipt that may arrive in two seconds, in four hours, or never; that may say "delivered" when the handset was off; that may arrive twice, out of order, or for a message the platform has no record of. Everything difficult here follows from that: what the acknowledgement at submit is allowed to promise, what happens when a submit times out with no answer, whether a status field or a fold over facts decides the truth, how much carrier capacity may be spent before a sender identity is suspended, and who is accountable when somebody texts STOP to a number four tenants share.

The design rests on one boundary. An accepted message is owned: it is validated, screened against consent and jurisdictional rules, and appended to a strongly consistent ledger before the API acknowledges anything, and the acknowledgement promises ownership rather than delivery. Everything after that point is disposable. A carrier attempt is a separate append-only record with its own route, segments, price and outcome, so a 503 is another attempt, a 429 is a park with a release time, a dead session is a resume, and a worker's death costs nothing. Delivery state is never written by the submit path and never stored as a mutable column: it is a fold over the attempt series and the receipts correlated to it, which is what makes a receipt arriving four hours late, twice, out of order, or after the body has been deleted a fact to append rather than a race to lose. Because the fold over "one attempt accepted, no receipt, deadline passed" is unknown, the platform's most unwelcome product requirement becomes structurally unavoidable rather than a matter of downstream discipline. Around that spine sit four things the platform does not control and must therefore model explicitly: carrier throughput windows, enforced by per-route leases because breaching one suspends a sender identity rather than merely rate-limiting it; traffic classes, because an OTP and a four-million-message campaign are different products sharing only the ledger; channel adapters, so that adding RCS is an adapter and a route family rather than a change to the state model; and a stateful session tier for long-lived carrier binds, which fit neither the elastic fleet above them nor the carriers below.

What it is, and what it is not

An owner of accepted messages and a recorder of what it triedA reporter of whether a handset received anything
A fold over attempts and receiptsA status column somebody updates
A platform where unknown is an answerA platform that rounds unknown to delivered
A courier inside other people's throughput windowsA system that retries until something works
The owner of regulated keywords and consent evidenceA pass-through that trusts the tenant's opt-out list
Three traffic classes that share only the ledgerOne queue with a priority field
A thin adapter contract with declared per-channel capabilityA lowest-common-denominator message that silently degrades
A keeper of receipts longer than the bodies they describeA store with one retention policy

The decisions that are the architecture

01The accepted message is owned; the carrier attempt is disposable

Two durable writes per attempt buy a system where every carrier failure is the same mechanism: appending a fact. The alternative pays by re-running what already succeeded.

ADR-01

02Delivery state is a projection, not a stored field

Out-of-order receipts are the normal behaviour of carrier networks, not an edge case. A column loses that race silently; a fold cannot.

ADR-02

03Accept before route

Submission never waits on a route, a session or a carrier, so the two fail separately and the submit SLO is achievable without any of them being healthy.

ADR-03

04Unknown is a first-class terminal state

A stored status always has a tempting default for "we heard nothing". A projection has none, so the honesty is enforced by structure rather than by discipline.

ADR-04

05The ambiguous submit is resolved by recorded per-route policy

No idempotency key means choosing between a duplicate code on a handset and a silent non-delivery. The choice differs by route and class, and is written onto the message.

ADR-05

06Uncorrelatable receipts are quarantined, never discarded

A receipt matching nothing is usually a platform bug and occasionally the evidence in a billing dispute. Dropping it destroys both.

ADR-06

07Per-route throughput leases, checked out in blocks

A carrier window is contractual: breaching it suspends a sender identity. Correctness here is worth a bounded overshoot and a lease that can be stranded.

ADR-07

08Three classes with a declared shedding order

Interactive, conversational and bulk get separate lanes and separate SLOs, and load is shed by refusing in a stated order rather than by making everything slower.

ADR-08

09Routing decides at attempt time

A route chosen at submit may be dead when the attempt runs. The price quoted at submit becomes an estimate, and that is said out loud in the API.

ADR-09

10A thin adapter contract with declared capability

Channels differ in ways tenants need to see. Hiding the differences makes the platform responsible for a translation nobody asked for.

ADR-10

11A dedicated stateful session tier

Long-lived binds with sequence windows and allowlisted egress do not belong in an elastic fleet where every scale-in is a teardown and an in-flight ambiguity.

ADR-11

12The platform owns regulated keywords

STOP on a shared number cannot be a tenant preference. The platform actions it, then tells the tenant it has already been actioned.

ADR-12

13Jurisdictional rules are versioned data and fail closed

A rule change must not need a deployment, and a message the platform is unsure it may send is not sent. Everything else degrades.

ADR-13

14Receipts outlive the bodies they describe

Content retention is a privacy decision; receipt retention is an evidence decision. Collapsing them means disputes arrive after the proof has gone.

ADR-14

15One rated usage event per attempt

A fallback hop, a re-route and a multi-segment body each cost money independently, so the message is the wrong unit to rate.

ADR-15

16Route eligibility is earned from measured delivery

Contracted quality is a sales document. A route with no measured outcome cannot stay eligible, because absence of receipts is itself a signal.

ADR-16

Why this should still be right in ten years

SMPP, the aggregator of the month, Spanner, Bigtable, the channel that replaces RCS and probably the regulator's current rulebook will all be replaced inside this platform's life. These five properties should survive the replacement, and each is a statement about where something lives rather than about what it is built from.

Truth is a fold over facts, not a field

Any system whose outcome is reported asynchronously by a party it does not control faces the same race, and the race gets worse as the number of reporting parties grows. A projection over append-only facts is indifferent to arrival order, which is the one property that does not degrade with scale, with a new channel, or with a carrier whose receipts are worse than the last one's.

The unit of durability is the promise, not the attempt

What the platform promised a caller and what it tried on their behalf are different lifetimes with different owners. Keeping them separate is why a retry, a park, a re-route and a worker death are one mechanism rather than four special cases, and it will still be why when the protocol underneath has changed twice.

What the platform cannot know is part of the product

Naming the unknown rather than rounding it away is a design stance, not a feature. It survives because the alternative compounds: once a status field has been allowed a comfortable default, every downstream report, SLO and routing signal inherits the lie, and no later component can recover the distinction.

Somebody else's capacity is a design input

The platform will always be spending throughput it rents under terms it did not set. Modelling that as a lease with a release time, rather than as an error to retry, keeps pacing a normal state and keeps the platform's relationship with its suppliers intact — whoever the suppliers are by then.

Evidence is retained on its own clock

The retention of content, of the record of what was done with it, and of the proof of permission are three different obligations with three different time horizons and three different regulators. A system that gives them one policy will be wrong about at least two, and the mistake is only discovered when somebody asks.

Non-functional targets

Every target below is a stated assumption, sized for the operating context in the evidence note and chosen so a reviewer can argue with a number rather than with a sentiment. Each names the view where the mechanism that delivers it is drawn.

QualityTargetHow it is metView
Submit availability ≥ 99.99% monthly Stateless submit across zones, one durable ledger write, no dependency on any route, session or carrier being healthy 13
Receipt and inbound ingest availability ≥ 99.99% monthly Verify, make durable, acknowledge — correlation happens afterwards, so ingest never waits on the ledger being queryable 14
Consent evaluation availability ≥ 99.99% monthly Strongly consistent store read on every submit, cached with a bounded staleness ceiling, failing closed past it 21
Submit acknowledgement latency p95 ≤ 150 ms, p99 ≤ 400 ms One synchronous ledger commit; validation, sender and consent checks are local reads or cached reads 13
Interactive time to carrier acceptance p50 ≤ 700 ms, p95 ≤ 2 s Dedicated interactive lane, per-route lease taken locally from a checked-out block, no cross-region hop 15
Receipt visibility p95 ≤ 2 s from carrier receipt Stream-driven correlation and fold, with the projection written ahead of the callback dispatch 14
Throughput 8,000 msg/s steady, 40,000/s for 10 min Independent scaling of submit, scheduling, attempt and ingest; ledger partitioned by tenant and time 17
Durability of accepted messages RPO 0, zero lost Synchronously replicated ledger; the acknowledgement is issued only after commit 11
Projection recovery RTO 2 h to rebuild State is a fold, so the projection is disposable; status queries degrade rather than fail during a rebuild 10
Duplicate handset deliveries ≤ 1 per 10,000,000 accepted Client-reference idempotency at submit, a single attempt key per carrier call, and a default of not re-attempting an ambiguous submit 16
Consent correctness zero sends after withdrawal, outside 30 s Inbound keyword interception writes consent before the tenant is notified; propagation bound is stated, not implied 06
Fairness interactive queue age p99 ≤ 3 s Class lanes first, then per-tenant share capped at 40% of a shared route while others have work queued 19
Cost platform ≤ 8% of carrier segment cost Ledger write path held to ≤ 35% of platform cost; dormant tenants and idle routes near zero 19

Scope

In scope

  • Message submission: validation, client-reference idempotency, segment counting, time-to-live, traffic class, batch and dry-run
  • Destination normalisation, sender-identity registry and jurisdictional sender rules as versioned data
  • Routing across carrier and aggregator routes with cost, quality and pinned policies, per-route breakers and automatic interactive failover
  • The append-only attempt series, retryability classification, bounded retry inside the time-to-live, and a recorded per-route ambiguous-submit policy
  • Receipt ingestion: verification, dedup, correlation, quarantine of the uncorrelatable, and the monotonic state fold including unknown
  • Status by pull and by signed callback with retry and a dead-letter path, reconstructible entirely from pull
  • Inbound messages, per-route dedup, tenant routing by sender identity, and regulated keyword interception with immediate consent enforcement
  • Consent records and jurisdictional compliance on the submit path, failing closed, with append-only history and per-destination evidence
  • Per-tenant entitlement, traffic-class scheduling, cross-tenant fairness, credit checks and suspension
  • Channel adapters for SMS, MMS and at least one rich channel, behind one internal contract, plus the stateful carrier session tier
  • One rated usage event per attempt, and the audit log that records every configuration and access event

Explicitly out of scope

  • Metering, rating tables and invoicing — this platform's output is the rated usage event; see usage-metering-and-billing-pipeline
  • Generic outbound webhook fan-out to tenant systems; see webhook-delivery-service
  • Application notification orchestration and end-user channel preference logic; see event-driven-notification-platform
  • Standalone request deduplication as a shared service; see idempotency-and-dedup-service
  • Voice and video channels, and the number-provisioning marketplace — phone numbers are an inventory this platform reads
  • Campaign authoring, audience segmentation and marketing analytics
  • The tenant's own identity provider, and end-user account recovery inside the tenant's product

What a four-week prototype should prove

The prototype's job is to falsify the three claims the design rests on: that a fold over attempts and receipts is cheap enough to answer status queries at rate, that per-route lease enforcement holds a contractual window across an elastic fleet without a cross-region hop, and that the ambiguous-submit default produces fewer bad outcomes than re-attempting. Everything else can be adjusted; if any of those three is wrong, the boundary in ADR-01 has to move.

  1. Two routes in two jurisdictions through two different providers, one SMPP and one HTTP, because a single provider hides every difference the adapter contract exists for
  2. A receipt generator that is deliberately hostile: late by hours, duplicated, out of order, occasionally for an unknown reference, and silent for a configurable share of traffic
  3. The full fold implemented against the ledger with no status column anywhere, and status queries measured at 10x the expected rate while the projection is rebuilt
  4. A contractual window of 300 messages per second enforced across a fleet that scales from 2 to 40 workers during the test, with overshoot measured rather than assumed
  5. An ambiguous-submit harness that drops responses after the carrier has accepted, so the duplicate rate of the re-attempt policy can be compared with the unknown rate of the default
  6. One interactive tenant and one bulk tenant on the same route, with the bulk tenant submitting four million messages while interactive queue age is measured
  7. An inbound STOP delivered twice, out of order, on a sender identity shared by two tenants, with the consent write and both tenants' subsequent submits observed
  8. Deliberately not in the prototype: the console, more than two channels, billing integration, and any form of campaign tooling
  • Twenty-five thousand receipts a second are replayed against the fold while the Bigtable projection is rebuilt from the ledger, with status queries served throughout at ten times the expected rate: every message's final state matches the state computed from the same facts in arrival order, and no query fails
  • A 'delivered' receipt generated at 10:00 arrives after an 'undelivered' generated at 09:58, four hours after the message's deadline and after its body has been purged: the message is already unknown, the late receipt is retained as evidence, and no transition out of the terminal state occurs
  • A route with a contracted window of 300 messages per second is driven while the attempt fleet scales from 2 to 40 workers and back twice: measured peak throughput never exceeds the window by more than the configured block size, and stranded capacity from killed workers is visible as a throughput dip rather than as a breach
  • The carrier harness accepts a submission and then drops the response: the ambiguous outcome is appended, the per-route policy applies and is recorded on the message, and the resulting duplicate rate under a re-attempt policy is measured against the unknown rate under the default so the ADR-05 trade is argued with numbers
  • A bulk tenant submits four million messages on the same route a second tenant is using for login codes: interactive queue age stays within its budget throughout, and under induced overload bulk is refused before conversational and conversational before interactive
  • An SMPP bind is killed uncontrolled with forty messages in flight, then deliberately drained with forty in flight: the uncontrolled case produces exactly forty ambiguous attempts resolved by policy, and the drained case produces none
  • A STOP arrives twice, out of order, on a sender identity shared by two tenants: one consent event is written, one mandated confirmation is sent, both tenants' subsequent submits are refused with the consent version cited, and messages already accepted when it landed are counted and reported rather than silently delivered
  • A receipt arrives carrying a provider reference the platform never issued: it is quarantined with its payload, the quarantine rate-of-change alert fires, and after a simulated correlation fix the window is replayed and the affected folds recomputed

Open risks, carried rather than hidden

RiskIf it landsResponse
The ledger write path is the platform's cost and scale ceiling Two synchronously replicated writes per attempt at 8,000 messages a second with retries and fallbacks is the hottest path in the system; if it exceeds its stated share of per-message cost the design is uneconomic rather than merely expensive Price it explicitly as a reported metric per message rather than per month, hold it to the 35% share in the NFRs, and treat batching of intent writes as the first optimisation to evaluate — never by weakening the intent-before-call ordering
The ambiguous-submit default is a judgement a real customer may reject Not re-attempting protects a recipient from a duplicate code and accepts a silent non-delivery, which for a bank's login traffic may be exactly the wrong trade Keep it a per-route and per-class policy with the choice recorded on each message, publish the measured duplicate and unknown rates per route, and let a tenant's traffic class select the other default where they can defend it
Unknown-state share will be read as a platform defect Routes that never report receipts inflate the unknown rate for reasons outside the platform, and the honest number is a worse-looking number than a competitor's rounded one Report unknown share per route alongside the carrier's own reporting behaviour, make it a routing input so bad reporters lose traffic, and state the distinction in the product rather than in a support article
Carrier egress allowlists make regional expansion a correspondence exercise Standby binds in a second region need their source addresses approved by every carrier before failover works, on the carrier's timetable, not the platform's Treat the allowlist as a tracked dependency per route with its own state, refuse to count a region as a failover target until binds there have carried real traffic, and rehearse on a schedule rather than on an incident
Consent on shared sender identities is a liability concentrator One STOP on a number four tenants share must silence all four, and the platform — not the tenant — will be asked why it did not Keep regulated keywords platform-owned and the consent history append-only and queryable per destination, publish the 30 s propagation bound as a commitment, and make in-flight messages after a withdrawal a reported number rather than an unexamined gap
The quarantine of uncorrelatable receipts becomes a queue nobody works A rising quarantine rate is the earliest signal of a correlation defect and the easiest dashboard to stop reading, and the window in which a billing dispute can be answered closes quietly Give the quarantine an owner and an age distribution rather than a count, alert on the rate of change rather than the level, and require a named cause before any quarantined receipt ages out

Programmable Messaging API

Why every component and every technology on these 22 views is what it is, and what each choice costs.

Sixteen decisions, grouped by the question they answer, each with the forcing question, the alternatives, what would flip the choice, and why it should outlast the technology it is realised on.

Status of this document. This is a design, not a report on a running system. Every rate, latency, ratio, window and retention figure is a stated assumption, chosen to be defensible and arguable rather than measured. The operating context assumed throughout is a multi-tenant messaging platform serving 40,000 active tenants across 190 jurisdictions and 9 channels, with roughly 220 carrier and aggregator routes and 6,000 concurrent protocol sessions, accepting 8,000 messages per second at steady state with a 5x burst for 10 minutes, ingesting 25,000 delivery receipts and 2,500 inbound messages per second, and carrying about 20 billion outbound messages a month on Google Cloud in two regions. Where a carrier protocol or a channel's own rules settle a question, the protocol is followed and the records say so rather than claiming credit for it; every record here is about something the protocols deliberately leave to the platform.

How to read a record

QuestionThe forcing question: why a decision was needed at all.
ContextThe requirement, the scale and the constraint that make it hard.
DecisionWhat this architecture does, stated so it can be checked.
How it is realised on the stackThe concrete mechanism: which component or package, configured how, in which cluster.
Options weighedChosen, rejected, deferred, or right elsewhere, with the reason for each.
ConsequencesWhat the choice buys and what it costs, both kept visible.
Choose differently whenThe conditions that would flip the decision for your system.
Why it holds up over timeWhat keeps the decision right as scale, staff and technology change.
LessonThe principle that transfers beyond this platform.

Decision map

The ownership boundary 3

The decision that defines the architecture, and the two that follow from it directly.

ADR-01The accepted message is the unit of durability; the carrier attempt is disposable ADR-02Delivery state is a monotonic projection over attempts and receipts, not a stored field ADR-03Accept before route: submission never waits on a route, a session or a carrier

Telling the truth about delivery 3

What the platform does when the carrier is late, silent, contradictory, or talking about a message nobody recognises.

ADR-04Unknown is a first-class terminal state, reported as such ADR-05The ambiguous submit is resolved by a recorded per-route policy, defaulting to no re-attempt ADR-06Receipts that correlate to nothing are quarantined, never discarded

Capacity the platform rents 3

Spending somebody else's throughput without losing the right to spend it, and deciding whose traffic waits.

ADR-07Per-route throughput is enforced by leases checked out in blocks, not by a central broker or local guesswork ADR-08Three traffic classes with separate lanes, separate SLOs and a declared shedding order ADR-09Routing decides at attempt time, which makes the price quoted at submit an estimate

Channels and sessions 2

How much of a channel's strangeness is hidden, and where the one stateful thing in the platform lives.

ADR-10A thin adapter contract, with per-channel capability declared rather than hidden ADR-11Carrier protocol sessions live in a dedicated stateful tier, not in the attempt fleet

Consent, jurisdiction and evidence 3

The rules the platform will be fined for, and the records that answer for them years later.

ADR-12The platform owns regulated keywords; the tenant owns marketing preference ADR-13Jurisdictional rules are versioned data that fails closed, and changes without a deployment ADR-14Receipts and consent outlive the bodies they describe, and store-no-body is a supported mode

Cost and operability 2

What a message actually costs, and how a route earns the right to carry one.

ADR-15One rated usage event per attempt, not per message ADR-16Route eligibility is earned from measured delivery, and absence of receipts is a signal

Technology by capability

Every component on the twenty-two views, why it was chosen on Google Cloud, what would have been used instead, and the decision record that explains the trade. The topic belongs to no cloud; Google Cloud was chosen as the practice's honest rotation, and the requirement document stays vendor-neutral throughout.

Open source This design
CapabilityChoiceOriginCredible alternativeWhy this oneRecord
Message ledger: append-only submissions and attempts, strongly consistent, RPO 0 Cloud Spanner, multi-region instance, interleaved attempt rows under the message, partitioned by tenant and time Google Cloud managed A globally distributed SQL database with per-row region placement, or a partitioned relational cluster with synchronous replicas Reading a message's whole history must be one local read, and the submit acknowledgement must wait on a committed replicated write; interleaving gives the first and synchronous replication gives the second without an application-level transaction ADR-01
Delivery-state projection: status by message id and by tenant client reference Bigtable, keyed by message id with a secondary row keyed by tenant and client reference Google Cloud managed A wide-column store or a document store with a secondary index, rebuilt from the ledger either way The projection is disposable and read-heavy at a rate the ledger should not serve; a key-value shape with predictable latency suits a fold result that is rebuilt rather than migrated ADR-02
Receipt and inbound streams, and the class lanes between submit and attempt Pub/Sub topics: one per traffic class, one for receipts, one for inbound Google Cloud managed A managed Kafka or a durable log with per-partition ordering The platform needs durability and independent scaling far more than it needs ordering, because the fold is order-independent by construction; separate topics give the class isolation that the shedding order depends on ADR-08
Per-route pacing against a contracted carrier window Cloud Tasks, one queue per route with its dispatch rate set from the route table, beneath Memorystore leases Google Cloud managed A token-bucket service in front of every attempt, or per-worker budgets with periodic reconciliation A managed queue with a configurable dispatch rate is a second, independent ceiling that survives a lease-store incident, which turns a potential contract breach into a throughput dip ADR-07
Route leases and route-quality counters Memorystore for Redis, one instance per region, atomic block check-out by script Google Cloud managed A self-hosted in-memory store per region, or a consensus store if exactness were required Single-key atomicity on one shard is exactly the guarantee a lease block needs and nothing more; the store is explicitly not authoritative, so its loss costs throughput rather than correctness ADR-07
Stateless submit, status, attempt execution and ingest workers Cloud Run services, scaled independently per concern Google Cloud managed Containers on a managed Kubernetes cluster, which is what the session plane uses anyway Four workloads with unrelated load curves need four scaling decisions, and none of them holds state; a request-scaled runtime makes the dormant-tenant cost floor near zero as the cost requirement demands ADR-03
Long-lived carrier protocol sessions with sequence windows GKE StatefulSet with stable identities, one pod per bind, drained individually Google Cloud managed Kubernetes Dedicated virtual machines per carrier, or sessions co-located in the attempt workers A bind is an addressable, individually recoverable thing with in-flight state; it needs stable identity and controlled termination, which is precisely what a request-scaled runtime cannot offer ADR-11
Carrier-approved static egress addresses Cloud NAT with reserved addresses per region, one range per region allowlisted by each carrier Google Cloud managed A NAT appliance or per-carrier VPN and interconnect where a carrier requires private connectivity Carriers allowlist source addresses by correspondence on their own timetable, so the platform needs a small, stable, reserved set per region rather than ephemeral egress ADR-11
Message bodies and media, with per-tenant keys and store-no-body mode Cloud Storage with CMEK per tenant, lifecycle rules per retention setting Google Cloud managed An object store with customer-managed encryption and object lifecycle policies — the shape is the same anywhere Content has a shorter and separately configurable life than the record of it, which requires a store that can be purged independently of the ledger ADR-14
Carrier credential custody, per-route scoping, tenant-managed keys Secret Manager for route credentials with Cloud KMS for key material, reached only through a custody service Google Cloud managed A secrets manager behind a hardware security module where a tenant requires key custody in hardware Attempt workers must hold no long-lived carrier material, so the exchange of workload identity for a short-lived single-route token needs a brokered boundary rather than mounted secrets ADR-11
Consent history, sender-identity registry, route table and jurisdictional rules Cloud Spanner, append-only consent history and versioned configuration tables, cached in Memorystore with a staleness ceiling Google Cloud managed A relational cluster with synchronous replicas and an in-memory cache in front Consent is read on every submit and must be strongly consistent; configuration is read a billion times and written rarely, so it belongs next to the reader with an explicit invalidation path ADR-13
Route quality aggregation and evidence analytics Dataflow streaming jobs into BigQuery, feeding the Memorystore quality store Google Cloud managed A stream processor into a columnar warehouse — any equivalent pair Delivery rate, receipt latency and unknown share are windowed aggregates over the receipt stream, and the same warehouse answers the regulator and the margin question ADR-16
Audit log retained independently of tenant settings Cloud Logging sink into a Cloud Storage bucket with Bucket Lock, plus BigQuery for query Google Cloud managed An append-only object store with a retention lock and a columnar query layer A seven-year audit obligation needs write-once retention the platform's own operators cannot shorten, which is a storage-level guarantee rather than an application one ADR-14
Edge: TLS, tenant credential check, WAF and anycast entry Global external Application Load Balancer with Cloud Armor Google Cloud managed Any anycast load balancer with a managed WAF in front of the submit tier Submit is a global API with a 99.99% target and hostile inbound endpoints beside it; anycast entry plus managed WAF keeps that off the application's critical path ADR-03
Build, provenance and deployment of services; separate path for rules and routes Cloud Build with Artifact Registry and Binary Authorization; Terraform for infrastructure; a reviewed data-publication path for the route table and rule set Google Cloud managed plus open-source Terraform Any CI system with image signing and admission control, and any reviewed configuration pipeline A jurisdictional rule change must take effect without a deployment, which means configuration needs its own reviewed path with the same scrutiny as code rather than a shortcut around the gates ADR-13
Hostile-receipt testing before staging A carrier simulator suite as a blocking build gate: late, duplicated, out-of-order, silent and unknown-reference receipts This design Contract tests against provider sandboxes, which do not reproduce the pathologies that matter Production cannot be asked to produce a four-hour-late duplicate receipt on demand, and those are precisely the inputs the fold and the quarantine exist for ADR-02

The decisions, and the alternatives that lost

The ownership boundaryThe decision that defines the architecture, and the two that follow from it directly.

ADR-01

The accepted message is the unit of durability; the carrier attempt is disposable

Accepted

A submitted message and a carrier call are both "the message" in ordinary speech. Which of them is the thing the platform durably owns, and what happens to the other one when it fails?

Context
Every failure in this class of system looks different at first glance and identical underneath. A carrier returns 503. A carrier returns 429. A session's bind drops with forty messages in flight. A worker is descheduled mid-submit. A route is withdrawn by its provider. If the unit of durability is the carrier call, each of these is its own recovery path with its own state, and the platform ends up with five partially-correct mechanisms that disagree about what happened. If the unit is the accepted message, and the carrier call is a separate disposable record attached to it, all five become the same operation: append a fact about an attempt, and let the message's state follow. The cost is not small and has to be faced rather than discovered — durability on the submit path means the API cannot acknowledge until a replicated write has committed, and every attempt pays at least two more writes, which at 8,000 messages a second with retries and fallback hops is the hottest and most expensive path in the platform.
Decision
An accepted message is appended to a strongly consistent, append-only ledger before the API acknowledges, and that acknowledgement promises ownership, not delivery. Each carrier call is a separate append-only attempt record under that message, carrying its route, segment breakdown, provider reference, submit outcome and the receipts later correlated to it. Nothing durable lives in a worker or a session. No attempt is made without a stable attempt key, and no attempt record is ever updated in place.
How it is realised on the stack
The ledger is Cloud Spanner, partitioned by tenant and time so no single tenant's campaign creates a hot partition, with the message row and the attempt rows in the same interleaved family so reading a message's full history is one local read. Message bodies and media go to Cloud Storage under per-tenant CMEK keys with their own, shorter retention, so the ledger stays small and a body deletion does not touch the delivery record. Attempt execution runs on Cloud Run with no local state: an executor reads the message, appends intent, calls the carrier through the session plane or a provider's HTTP API, and appends the outcome. A descheduled executor loses nothing, because the intent record plus the absence of an outcome is itself a complete, resumable statement of what happened.
Options weighed
  • ChosenAccepted message durable, attempt disposable: Every carrier failure becomes one mechanism; pays two replicated writes per attempt and makes the ledger the platform's scaling constraint
  • RejectedThe carrier call is the unit, retried from the start on failure: Dramatically simpler and needs no ledger, but re-running a succeeded step means a second message to a human; for OTP traffic that is a wrong answer, not a slow one
  • RejectedMessage durable, attempts kept only in a log for diagnostics: Cheaper, and makes "what did we try" unanswerable at exactly the moment a tenant or a carrier disputes a charge
  • Right elsewhereQueue-as-state: the message lives in the broker until delivered: Right for a platform with one channel, one route and no billing dispute to settle; here it cannot represent a fallback chain or two attempts at different prices
Consequences
What it buys
  • Retry, park, re-route, session loss and worker death are one operation — appending a fact — rather than five recovery paths that can disagree
  • A fallback chain is expressible without special handling: each hop is another attempt with its own cost and its own terminal state
  • "What exactly happened to this message" is a single local read, which is what makes developer self-service diagnosis possible at all
  • Every charge is attributable to an attempt, which is what makes a carrier billing dispute winnable
What it costs
  • Two synchronously replicated writes per attempt make the ledger the platform's hottest write path and its primary scaling constraint
  • The API cannot acknowledge faster than a replicated commit, which sets the floor under the submit latency budget
  • Storage grows with attempts rather than messages, so a route with a poor acceptance rate costs more to record as well as to use
Choose differently when
If the platform only ever served one traffic class where a duplicate is harmless — server-to-server notifications with their own dedup, say — the ledger stops earning its cost and retry-from-the-start becomes the right answer. It would also flip if a carrier ecosystem emerged with universal, honoured idempotency keys: with the carrier able to answer "did you already take this?", the attempt record stops being the only way to know.
Why it holds up over time
The separation is between what was promised to a caller and what was tried on their behalf, and those have different lifetimes in any system that acts for somebody through a third party. That is true of a messaging platform, a payment orchestrator and a shipping integrator alike, and it stays true when the protocol underneath is replaced. The cost of the separation is paid per operation and the benefit grows with the number of distinct failure modes, which only ever increases as channels and providers are added.
LessonWhen several failures look different but recover the same way, the unit of durability is probably drawn in the wrong place. Move it up until they collapse into one mechanism, then price what that costs on the hot path and say so out loud.
Shown on views02 08 11 13
ADR-02

Delivery state is a monotonic projection over attempts and receipts, not a stored field

Accepted

Twenty-five thousand receipts a second arrive late, duplicated and out of order. Does the platform keep a status column and update it, or compute status from the facts?

Context
A status column is the obvious design: fast to query, trivial to reason about, and one row per message. It is also a race the platform loses quietly. Carrier receipts are not ordered: a "delivered" generated at 10:00 can arrive after an "undelivered" generated at 09:58, from different parts of a carrier's estate, hours after the message's deadline, and sometimes twice. With a column, the last writer wins, which means the truth depends on network scheduling. Worse, a column needs a default for the case where nothing has arrived, and there is no honest default — writing "sent" implies more than is known and writing "failed" implies less. A fold over the attempt series and the correlated receipts has no such default: the fold of "one attempt accepted, no receipt, deadline passed" is unknown, and no code had to decide that.
Decision
The state model is accepted, queued, sent, then one of delivered, undelivered, expired or unknown, and it is computed as a monotonic fold over the message's attempts and the receipts correlated to them. Status is never written by the submit path. A transition out of a terminal state is forbidden, whatever a late receipt says. The served projection is a rebuildable read model, and the ledger plus the receipt stream remain the only authority.
How it is realised on the stack
A stream processor reads the receipt stream from Pub/Sub, correlates each receipt to an attempt by provider reference, appends it, recomputes the fold for that message and writes the result to a Bigtable projection keyed by message id, with a secondary row keyed by tenant and client reference so a tenant can query with their own identifier. The fold is a pure function of the appended facts, so replaying the stream reproduces the projection exactly. A TTL sweeper appends an explicit expiry fact when a deadline passes with no conclusive receipt, which is what makes unknown a derived value rather than an absence. Rebuilding the whole projection from the ledger is a batch job with a two-hour target, during which status queries are served from a lagging projection rather than failing.
Options weighed
  • ChosenStatus as a monotonic fold, served from a rebuildable projection: Correct under any arrival order and has no dishonest default; status reads become reads of a cache that can lag and must be rebuildable
  • RejectedA mutable status column updated by each receipt: Simplest and fastest, and silently corrupts state under out-of-order receipts, which are normal carrier behaviour rather than an edge case
  • RejectedA status column plus a timestamp guard, rejecting older receipts: Fixes ordering using carrier clocks, which are not trustworthy across providers, and still needs a default for silence
  • DeferredCompute status on read, directly from the ledger, with no projection: Removes a store and the rebuild problem; viable at a fraction of this read volume, and worth revisiting if status query rates stay low
Consequences
What it buys
  • A receipt arriving four hours late, twice, or after the body has been deleted is a fact to append rather than a race to lose
  • Unknown appears in the API, the dashboard, the SLO and the route-quality signal without anyone having chosen to put it there
  • The projection is disposable, so a schema change to the read model is a rebuild rather than a migration
  • Replay is exact, which makes a correlation bug correctable after the fact rather than permanent
What it costs
  • Every status query reads a projection that can lag, and the lag has to be a monitored signal rather than an assumption
  • Rebuilding at 20 billion messages a month must complete in hours, which constrains how expensive the fold may be
  • Two stores hold overlapping information, and a reviewer will reasonably ask why
Choose differently when
If carriers converged on ordered, exactly-once receipt delivery with trustworthy timestamps, the ordering argument would weaken enough for a guarded column to be defensible. A collapse in read volume would also flip it: at low rates, computing from the ledger on read removes the projection entirely, which is why that option is deferred rather than rejected.
Why it holds up over time
The property that lasts is that arrival order is not under the platform's control and never will be, because the reporting parties are external and multiplying. A fold is indifferent to order; a column is not, and no amount of later engineering recovers information a lost write destroyed. The argument strengthens with every channel added, which is the opposite of how most simplifications age.
LessonIf a field has no honest default, it should not be a field. Compute it, and the uncomfortable answer appears by itself instead of being someone's decision to avoid.
Shown on views02 10 12 14
ADR-03

Accept before route: submission never waits on a route, a session or a carrier

Accepted

Should the submit call choose a route and confirm the carrier will take the message before acknowledging, so the tenant learns about problems immediately?

Context
There is a real argument for routing at submit: the tenant finds out at once that no route exists, the price quoted is the price charged, and there is no queue to explain. The argument fails on availability arithmetic. Submit is required to hold 99.99% monthly; routes, sessions and carriers are third-party components with no such guarantee, and 220 of them. Coupling the submit path to any of them makes the platform's headline availability the product of everybody else's, and makes a single carrier's bad afternoon look like an outage of the API. It also inverts the latency relationship that matters: a tenant's HTTP call is in front of a user, and a carrier's acceptance is not.
Decision
Submission validates what is knowable without a carrier — destination, sender identity, jurisdictional sender rules, body limits, template approval, quota, credit and consent — appends the message, and acknowledges. Route selection, lease acquisition, adaptation and the carrier call all happen afterwards, asynchronously, and their failures are message states rather than API errors. The only synchronous external-ish dependency left on the submit path is the consent read, which is the platform's own store.
How it is realised on the stack
Cloud Run submit services behind a global load balancer with Cloud Armor; validation and sender-rule checks read a Memorystore-cached snapshot of the registry with a bounded staleness ceiling; consent is a strongly consistent Spanner read; the ledger append is the one durable write; the message is then published to a Pub/Sub lane topic for its traffic class. A dry-run endpoint gives tenants the thing they lose by this decision — resolved route, segment count, estimated price and compliance verdict — without sending anything, so the information is available when it is wanted rather than imposed on every call.
Options weighed
  • ChosenAccept first, route later, with a dry-run for pre-flight answers: Submit availability is independent of 220 third parties; the price returned at submit becomes an estimate and queueing must be explained to tenants
  • RejectedRoute and confirm carrier acceptance synchronously: Truthful pricing and immediate errors, bought by making the API's availability the product of every carrier's and its latency the carrier's latency
  • RejectedHybrid: synchronous for interactive, asynchronous for bulk: Two submit semantics in one API, and the class that most needs availability would be the one coupled to carriers
  • DeferredSynchronous route selection without carrier confirmation: A middle position that makes the quoted price firmer at modest cost; blocked for now by ADR-09, which wants route health evaluated at attempt time
Consequences
What it buys
  • The submit SLO is achievable without any route, session or carrier being healthy
  • A carrier outage is a queue and a set of message states, not an API error rate
  • Load from a four-million-message campaign lands on the scheduler, not on the API's latency profile
  • Validation rejections stay fast and specific, because they need nothing external
What it costs
  • The segment count and price returned at submit are estimates, and a re-route or fallback hop can exceed them
  • "Accepted" has to be explained, and some tenants will read it as "delivered" however it is documented
  • A message can be accepted and then fail for a reason that was knowable at submit time, such as a route having just been withdrawn
Choose differently when
A single-channel, single-route platform with one carrier contract and no fallback could afford synchronous confirmation, because there is nothing to be independent of. It would also flip for a product whose tenants genuinely cannot tolerate an estimated price — a wholesale reseller, say — where firm pricing at submit is worth coupling to.
Why it holds up over time
Separating the promise to accept from the attempt to deliver is the oldest pattern in integration and the one that keeps a platform's availability its own. What changes over a decade is the number of third parties, which only grows; the value of not being coupled to them grows with it.
LessonBefore coupling a path to a dependency, multiply the availability targets. If the product is below the path's own SLO, the coupling is the decision being made, whatever else the design discussion is about.
Shown on views02 07 13

Telling the truth about deliveryWhat the platform does when the carrier is late, silent, contradictory, or talking about a message nobody recognises.

ADR-04

Unknown is a first-class terminal state, reported as such

Accepted

A message was accepted by a carrier, no receipt ever arrived, and the deadline has passed. What does the platform tell the tenant?

Context
Some routes never report. Some report for 90% of traffic. Some report only failures. A platform that insists on a binary answer must therefore invent one, and the invented answer is always the optimistic one, because the sender wants to believe it and nobody downstream can contradict it. The consequences compound: an inflated delivery rate becomes the route-quality signal that drives routing, so the worst-reporting routes look the best; the SLO measures a number the platform made up; and the developer whose customer did not receive a code is told the message was delivered, which converts a solvable support question into a dispute about the platform's honesty.
Decision
Unknown is a terminal state in the model, reached when a message's time-to-live elapses with no conclusive receipt. It is returned by the API, shown in the console, counted in the SLOs, reported per route, and used as a route-quality input. It is never collapsed into delivered or undelivered, and the absence of a receipt is never treated as success.
How it is realised on the stack
A TTL sweeper appends an explicit expiry fact to the ledger when a deadline passes with no conclusive receipt, so unknown is derived by the same fold as every other state rather than inferred by a query. Unknown share is aggregated per route in BigQuery and fed back into the Memorystore route-quality store, so a route that stops reporting loses weight in the least-cost and best-quality policies. The tenant-facing vocabulary distinguishes three things explicitly: delivered, undelivered with a reason, and unknown with the attempt trail that produced it.
Options weighed
  • ChosenUnknown as a terminal state, reported and used as a signal: Honest, and produces a worse-looking delivery rate than competitors who round; requires the distinction to be explained in the product
  • RejectedTreat no-receipt as delivered after the deadline: What the sender wants to hear, and it poisons the routing signal, the SLO and the support conversation at the same time
  • RejectedTreat no-receipt as undelivered after the deadline: Pessimistic in the other direction: triggers tenant retries for messages that did arrive, which means duplicates on handsets
  • RejectedPer-route configuration of the assumed outcome: Makes the platform's reported truth a configuration value, which is the same lie with an audit trail
Consequences
What it buys
  • Routing is driven by observed delivery rather than by an assumption that flatters bad reporters
  • The developer's "nothing arrived" conversation starts from a platform answer rather than a platform denial
  • The SLO measures something real, which means it can be improved rather than defended
  • Carriers with poor reporting become visible as a commercial problem rather than a silent quality tax
What it costs
  • The headline delivery rate is lower than a competitor's rounded figure, and sales will have to argue the point
  • Unknown share varies by route for reasons outside the platform's control and will be read as a platform defect
  • Tenants need a documented position on what to do with an unknown, which is a product decision as much as an API one
Choose differently when
Nothing short of universal, reliable carrier reporting would flip this, and that is a commercial change rather than a technical one. If a jurisdiction's regulator mandated receipt reporting and it were enforced, unknown would shrink to a rounding error there and the state could in principle be retired for those routes — but the model would keep it, because the first new channel would need it again.
Why it holds up over time
Any platform that acts through parties it cannot observe will have outcomes it cannot determine, and the share of them grows as the number of parties grows. Naming that category is a stance that does not depend on the technology; abandoning it is irreversible, because once a comfortable default exists every downstream consumer inherits it and the distinction cannot be recovered.
LessonA system should be allowed to say it does not know. Removing that option does not create knowledge — it relocates the guess to the place where it is least visible and most expensive.
Shown on views12 14 19
ADR-05

The ambiguous submit is resolved by a recorded per-route policy, defaulting to no re-attempt

Accepted

An attempt times out with no response and the carrier offers no idempotency key. The message may have been delivered, dropped, or never received. Does the platform try again?

Context
This is the sharpest trade in the platform and it has no answer that is right everywhere. Re-attempting risks a second message to a person — two login codes, two appointment reminders — and a second charge the carrier will also levy. Not re-attempting risks a silent non-delivery that the tenant will attribute to the platform. Probing is only available where a route supports querying a provider reference, which most do not. The right answer genuinely differs: for an OTP, a duplicate is confusing but a missing code blocks a login; for a marketing message, a duplicate is a complaint and a miss is nothing. A single platform-wide rule will therefore be wrong for a large share of traffic whichever way it is set.
Decision
An attempt that times out with no response is recorded as an ambiguous outcome, which is a fact rather than a failure. Whether to re-attempt is a per-route policy with a per-traffic-class override, and the policy applied is written onto the message so the decision is auditable. The default is not to re-attempt on a route that cannot be queried. Where a route supports querying by provider reference, the policy is to probe before deciding.
How it is realised on the stack
The session plane records the ambiguity when a sequence window times out, and the attempt's outcome field carries it explicitly rather than being left empty. The route table holds three fields per route — queryable, default ambiguity policy, and per-class overrides — as versioned control-plane data, so the policy can change without a deployment. The executor reads the policy, acts, and appends the policy version it applied. Ambiguous-submit rate is a reported metric per route, and the measured duplicate rate of re-attempting routes is published beside the unknown rate of the others, so the trade is argued with numbers rather than with intuition.
Options weighed
  • ChosenPer-route policy with class overrides, default no re-attempt: Protects the recipient by default and lets a defensible tenant choose otherwise; needs the policy recorded per message or the decision is unauditable
  • RejectedAlways re-attempt: Maximises apparent delivery and puts duplicate codes on handsets and duplicate charges on invoices
  • RejectedNever re-attempt, no override: Simple and defensible, and wrong for interactive traffic where a missing code is worse than a duplicate
  • Right elsewhereAlways probe: The correct answer on a route estate where querying is universal; here most routes cannot be queried, so it would be a policy that mostly does not apply
Consequences
What it buys
  • The default protects the person holding the handset, which is the actor with no relationship to the platform
  • The trade is visible and arguable per route, with measured duplicate and unknown rates beside it
  • Every message carries the policy that decided its fate, so a dispute three weeks later has an answer
  • Routes that support querying get a materially better outcome, which gives carriers a reason to offer it
What it costs
  • Two tenants on the same channel can legitimately see different behaviour, which is harder to document than one rule
  • The default accepts silent non-deliveries, and some of them will be login codes
  • Policy is control-plane data that changes behaviour, so it needs the same review as code
Choose differently when
Universal honoured idempotency keys across carrier protocols would remove the ambiguity and the decision with it. Short of that, measured evidence could flip the default: if the duplicate rate from re-attempting turns out to be far lower than the unknown rate it prevents, the interactive default should change — which is exactly what the prototype is asked to measure.
Why it holds up over time
Ambiguity under timeout is inherent to any remote call without an idempotency mechanism, and will outlive every protocol in this design. What lasts is the shape of the answer: make the ambiguity a recorded fact, make the resolution a policy rather than a constant, and record which policy ran. That remains correct whether the right default moves or not.
LessonWhen a trade-off genuinely differs by context, the architecture's job is to make the choice explicit, per context, and recorded — not to pick a side and hard-code it where nobody can see it.
Shown on views13 16 19
ADR-06

Receipts that correlate to nothing are quarantined, never discarded

Accepted

A receipt arrives carrying a provider reference the platform has no record of. Is it noise to drop, or data to keep?

Context
The instinct is to drop it: an unknown reference looks like a carrier's mistake, a test from their side, or a replay. In practice it is usually the platform's own fault — a correlation key written wrongly, an attempt record that failed to commit, a provider that changed its reference format, a route migrated between accounts — and each of those is a defect that the dropped receipts were the only evidence of. The remainder matter commercially: a carrier invoice lists messages the platform cannot match, and without the receipts there is nothing to dispute with. Dropping is also irreversible in a way that keeping is not.
Decision
An uncorrelatable receipt is appended to the receipt stream and written to a quarantine store with its full payload, its route, and the reason correlation failed. It is never discarded. The quarantine has an owner, an age distribution and an alert on its rate of change, and a quarantined receipt may age out only after a named cause has been recorded. The data model permits this directly: the receipt's attempt reference is nullable by design.
How it is realised on the stack
Correlation runs on Cloud Run workers reading the receipt stream; a failed match writes to a Bigtable quarantine table partitioned by route and day, retained 30 days, with the raw payload preserved verbatim. Rate of change rather than level is alerted, because a steady background trickle from one provider is a known condition and a step change is a new defect. Quarantined receipts are replayable: once a correlation bug is fixed, the window can be re-processed and the folds recomputed, because state is a projection rather than a stored field.
Options weighed
  • ChosenQuarantine with full payload, owner and rate alerting: Preserves the evidence of the platform's own defects and of billing disputes; costs a store and, more importantly, somebody's attention
  • RejectedDrop with a counter: Cheap, and it converts the earliest signal of a correlation defect into a number nobody can act on
  • RejectedLog the payload and rely on log search: Looks equivalent and is not: logs are retained on a different clock, are not replayable into the fold, and are rarely retained for the 13 months a dispute needs
  • RejectedReject at the endpoint with an error to the carrier: Teaches carriers to retry or to stop sending, and loses the payload either way
Consequences
What it buys
  • Correlation defects are discoverable and, because state is a projection, correctable after the fact by replay
  • A carrier invoice line the platform cannot match has evidence behind it
  • A provider changing its reference format shows up as a quarantine step change rather than as a silent drop in delivery rate
  • The nullable reference in the model makes the intent explicit to anyone reading the schema
What it costs
  • A store, a retention policy and a named owner for data that is, by definition, not useful yet
  • A quarantine with no owner becomes a dashboard nobody reads, which is worse than a counter because it looks like diligence
  • Raw payloads may carry content, so the quarantine needs the same classification and encryption as the body store
Choose differently when
If the quarantine rate stayed at zero for a long period across all routes, the store could be reduced to sampling. It would also change if receipts were self-describing and signed in a way that let the platform prove a reference was never issued by it — then an unmatched receipt really would be somebody else's problem.
Why it holds up over time
Any system correlating externally generated events to internal records will have events it cannot place, and the rate of them is a direct measure of its own correctness. Keeping them is how the system learns about itself. That is independent of protocol, channel and decade, and the cost of keeping them stays small relative to what they reveal.
LessonData a system cannot explain is the most valuable data it has, because it is a measurement of the system's own understanding. Discarding it is discarding the only instrument pointed inward.
Shown on views12 14 19

Capacity the platform rentsSpending somebody else's throughput without losing the right to spend it, and deciding whose traffic waits.

ADR-07

Per-route throughput is enforced by leases checked out in blocks, not by a central broker or local guesswork

Accepted

A carrier's window of 300 messages per second is contractual, and breaching it suspends a sender identity. How is that enforced across a fleet that scales from two workers to forty?

Context
This is not a rate limit in the ordinary sense. Exceeding an API's quota earns a 429; exceeding a carrier's contracted window earns a throttled or suspended sender identity, a commercial conversation, and sometimes a jurisdiction the platform cannot send into for a week. So the enforcement has to hold across an elastic fleet, and it has to hold while sitting inside an interactive latency budget of two seconds end to end. A central broker every attempt passes through is correct and puts a network hop, and a new single point of failure for all sending, on the hot path. Per-worker local budgets reconciled periodically are fast and overshoot by exactly whatever the reconciliation window allows, which is the wrong variable to be imprecise about. Leases checked out in blocks are the compromise, and their failure mode is a crashed worker holding capacity nobody can spend.
Decision
Each route's window is divided into leases. A worker checks out a block of permits for a route, spends them locally with no further coordination, and returns or forfeits the remainder. Block size and lease duration are per-route configuration, sized so that the worst-case stranded capacity is a stated fraction of the window rather than an unknown. Lease state is regional and explicitly not authoritative for correctness: losing it costs throughput, never a duplicate or a lost message.
How it is realised on the stack
Leases live in Memorystore for Redis, one instance per region, with block check-out as a single atomic script so a lease is never double-issued. Cloud Tasks queues provide the pacing underneath, one queue per route with its dispatch rate set from the route table, which gives a second, independent ceiling that survives a lease-store incident. Leases are deliberately regional rather than shared across regions: a shared lease store would put a cross-region round trip inside the interactive budget, and would make a carrier window depend on inter-region connectivity. The split of a route's window between regions is configuration, reviewed like any other route data.
Options weighed
  • ChosenLeases checked out in blocks, regional, with a pacing queue underneath: No hop per attempt and a bounded, stated overshoot; a crashed worker strands a block until its lease expires
  • RejectedCentral lease broker on every attempt: Exact enforcement, bought with a hop inside the interactive budget and a component whose outage stops all sending
  • RejectedPer-worker local budgets with periodic reconciliation: Fastest, and the overshoot is whatever the window allows — unacceptable when the penalty is suspension rather than a 429
  • Right elsewhereOne sending worker per route, so the window needs no coordination: Right for a platform with a handful of routes; at 220 routes it is 220 singleton workloads and a availability problem per route
Consequences
What it buys
  • Attempt execution pays no coordination cost per message, which is what makes the interactive p95 reachable
  • Worst-case overshoot is a configured number rather than an emergent one, so it can be negotiated with a carrier
  • The pacing queue is a second independent ceiling, so a lease-store incident degrades throughput rather than breaching a contract
  • Regional leases keep a carrier window independent of inter-region connectivity
What it costs
  • A worker that dies holding a block strands that capacity until the lease expires, which shows as a throughput dip
  • Block size and lease duration are per-route tuning parameters, and getting them wrong is either overshoot or waste
  • A route's window must be split across regions in configuration, which is one more thing to get wrong during a regional event
Choose differently when
If a carrier offered a window enforced on their side with a clean, cheap backpressure signal rather than a penalty, local budgets would be sufficient and the lease machinery could go. Conversely, if sub-millisecond coordination became available in-region, a central broker's hop would stop mattering and exactness would be worth taking.
Why it holds up over time
Platforms will always be spending capacity they rent under terms they did not set, and the gap between an elastic fleet and a fixed external window will not close. Leasing a divisible right to spend, locally, with a bounded loss on failure, is the general answer; only the store and the latency numbers change.
LessonMatch the strength of a coordination mechanism to the cost of being wrong, not to the elegance of being exact. When the penalty is commercial rather than technical, a stated worst case beats an unbounded approximation and a hop you cannot afford.
Shown on views07 15 17 20
ADR-08

Three traffic classes with separate lanes, separate SLOs and a declared shedding order

Accepted

A tenant submits four million marketing messages while another tenant sends one login code. How does the platform decide what waits?

Context
Interactive and bulk traffic are different products that happen to share an API. A login code is worthless thirty seconds late; a campaign is fine arriving over four hours, and its value is in completing inside a window rather than in any individual message's latency. One queue with a priority field averages the two: the campaign's volume raises the code's queue age, and the code's urgency distorts the campaign's pacing. Under overload, a single queue has only one lever — everything slows — which is the worst available outcome, because it degrades the traffic that cannot tolerate degradation in order to protect traffic that could have waited.
Decision
Every submission declares a traffic class: interactive, conversational or bulk. The classes get separate queues, separate scheduling, separate route preferences and separate SLOs. Scheduling is by class first and fairness within class second, with no tenant taking more than 40% of a shared route's capacity while another tenant has work queued for it. Under overload the platform sheds by refusing in a declared order — bulk, then conversational, then interactive — rather than by extending latency across the board.
How it is realised on the stack
Separate Pub/Sub topics per class, with subscriber fleets scaled independently, so bulk backlog cannot occupy interactive consumer capacity. Bulk carries a route preference biased to cost and is paced to the tenant's stated completion window; interactive is biased to measured quality and gets first claim on lease blocks. Queue age is reported per class and per route and exposed to tenants, so a paced campaign is visibly paced rather than apparently stuck — which is the support cost this decision creates and must therefore answer. Class is required at submit rather than inferred, because inference from content or volume is wrong exactly when it matters.
Options weighed
  • ChosenThree declared classes, separate lanes, declared shedding order: Protects the traffic that cannot wait and makes overload behaviour predictable; tenants must classify honestly and pacing must be made visible
  • RejectedOne queue with a priority field: Simplest, and under load it degrades everything to protect nothing in particular
  • RejectedInfer the class from volume and content: Removes a field from the API and gets the answer wrong for the single urgent message inside a bulk submission
  • Right elsewherePer-tenant dedicated capacity instead of classes: Right for a small number of large tenants; at 40,000 tenants the cost floor per tenant makes it uneconomic
Consequences
What it buys
  • Interactive queue age holds regardless of concurrent bulk volume, which is the platform's core latency promise
  • Overload behaviour is a stated order of refusal rather than an emergent slowdown
  • Bulk can use cheaper routes and longer windows without compromising the interactive quality bar
  • Fairness is expressible per route, which is where contention actually happens
What it costs
  • Tenants can misclassify, deliberately or carelessly, and entitlement has to police it
  • Three lanes is three sets of capacity decisions and three SLOs to measure and defend
  • Pacing must be made visible, or every paced campaign becomes a support ticket about being stuck
Choose differently when
A platform serving only transactional traffic would not need the split, and one serving only campaigns would not either. Classes earn their complexity precisely because both share the ledger. If bulk were ever moved to a separate product with its own API, this decision would collapse back to a single lane.
Why it holds up over time
Mixing workloads with different value-of-latency on shared scarce capacity is a permanent condition of multi-tenant platforms. Declaring the classes and the shedding order is what makes behaviour under stress a design property rather than a discovery, and that reasoning survives any change in the queueing technology.
LessonDecide what to refuse before you are forced to. A system without a declared shedding order has one by default, and it is usually the order that protects the least important thing.
Shown on views07 15 19
ADR-09

Routing decides at attempt time, which makes the price quoted at submit an estimate

Accepted

Is the route chosen when the message is accepted, or when the attempt actually runs?

Context
Choosing at submit has real virtues: the price quoted at acceptance is the price charged, the decision is auditable against exactly one version of the route table, and a support engineer three weeks later has an unambiguous answer. It also binds a message to a route that may be withdrawn, throttled or silently failing by the time the attempt runs — which, for a bulk campaign paced over four hours, is not a corner case but the normal condition. Choosing at attempt time uses current health and current quality, and pays by making the submit-time price an estimate that a re-route or a fallback hop can exceed. The two are not equally wrong: a stale route produces a failed or unknown message, while an estimated price produces a conversation.
Decision
Route selection happens per attempt, against the route table version current at that moment, under a tenant-visible policy of least cost, best measured quality, or an explicitly pinned route. The attempt record carries the route chosen, the policy that chose it, the table version, and the alternatives considered. The submit response returns an estimated segment count and price, labelled as an estimate, and the dry-run endpoint gives a firm pre-flight answer for tenants who need one before sending.
How it is realised on the stack
The route table and prices live in Spanner and are cached in Memorystore with an explicit invalidation path, so a selection costs a local read. The selector reads the Memorystore quality store — delivery rate, receipt latency and unknown share per route, recomputed continuously from the receipt stream — and applies the tenant's policy. The alternatives considered are written onto the attempt as a short list with their scores, because a routing decision nobody can reconstruct is indistinguishable from a random one when a tenant asks why their message cost three times the quote.
Options weighed
  • ChosenSelect per attempt, record the decision and the alternatives: Uses current health, so a withdrawn route never strands a queued message; the submit price becomes an estimate and must be labelled as one
  • RejectedSelect at submit and bind the message to the route: Firm pricing and a single auditable table version, bought by sending queued messages down routes that have since died
  • RejectedSelect at submit, re-select only on failure: A reasonable middle that still binds the first attempt to stale health, which is the attempt that matters for interactive traffic
  • Right elsewhereLet tenants pin a route always: Right for a wholesale customer with their own carrier contracts; offered as a policy, not made the default
Consequences
What it buys
  • A route withdrawn or collapsed after submit never receives the queued traffic, which matters most for long-paced campaigns
  • Quality signals have somewhere to act, which is what makes the route lifecycle loop meaningful rather than decorative
  • Every routing decision is reconstructible from the attempt record, including what was not chosen
  • Interactive failover needs no operator, because the next attempt simply reselects
What it costs
  • The price returned at submit is an estimate, and a fallback or re-route can exceed it
  • Tenants can be charged different amounts for identical messages minutes apart, which needs explaining in the product
  • Storing alternatives considered on every attempt adds to the platform's hottest write path
Choose differently when
If route health were stable over hours rather than minutes — a small, reliable route estate under direct contract — selecting at submit would be the better trade, because firm pricing has real commercial value and the staleness risk would be negligible. A regulatory requirement to quote a firm price before sending would also flip it.
Why it holds up over time
Deciding late is correct whenever the information needed to decide well decays faster than the work waits. That relationship is a property of the dependency, not of the technology, and it holds for carrier routes, for capacity markets and for any supplier whose availability is measured in minutes.
LessonBind to a supplier as late as the product's promises allow. Early binding buys certainty about price and spends it on certainty about availability, and only one of those two is usually recoverable afterwards.
Shown on views05 09 20

Channels and sessionsHow much of a channel's strangeness is hidden, and where the one stateful thing in the platform lives.

ADR-10

A thin adapter contract, with per-channel capability declared rather than hidden

Accepted

How much of a channel's strangeness should the platform absorb — template approval, rich cards, media rules, conversation windows — and how much should the API expose?

Context
A thick adapter promises one message that works everywhere: the platform negotiates capability, degrades a rich card to text, substitutes an approved template, and the tenant sends the same payload to every channel. It is attractive until the degradation matters. A marketing message designed as a carousel, silently delivered as 160 characters, is a different product than the one the tenant authored, and the platform has quietly taken responsibility for an editorial decision. A thin adapter keeps a small common core — destination, sender identity, body or template reference, media reference — and makes the per-channel requirements explicit, visible fields that a tenant must satisfy. It leaks the carriers' inconsistencies into the API, which is honest and more work for the tenant.
Decision
The internal adapter contract is deliberately thin: submit an attempt, receive an acceptance or rejection, receive asynchronous receipts. Channel-specific requirements are first-class, declared, tenant-visible properties — template approval state, media constraints, conversation windows, capability probe results — and a fallback is an explicit tenant-declared chain where each hop is a separate attempt with its own cost and terminal state, never an invisible degradation.
How it is realised on the stack
Each channel is a Cloud Run adapter implementing the same three-method contract plus a capability declaration and a retryability map for its error codes. The message spine knows only messages, attempts and routes; adding a channel adds an adapter and a route family and touches neither the ledger schema nor the state model, which is the property the thin contract exists to preserve. Where a channel needs a probe — RCS capability, for instance — the probe result is recorded on the attempt, so a fallback to SMS is explicable rather than mysterious, and both hops appear in the rated usage.
Options weighed
  • ChosenThin contract, declared capability, explicit fallback chains: No silent degradation and no change to the spine per channel; tenants carry per-channel knowledge the platform could have hidden
  • RejectedThick adapter with automatic capability negotiation: A simpler API, and it makes the platform responsible for editorial decisions nobody asked it to make
  • RejectedLowest-common-denominator message only: Thinnest of all, and it makes rich channels unusable, which is most of the roadmap
  • Right elsewherePer-channel APIs with no common core: Right for a platform selling one channel deeply; here it forfeits the fallback chain and multiplies the tenant's integration work
Consequences
What it buys
  • Adding a channel is an adapter and a route family, with no change to the ledger, the state model or the spine
  • A fallback is two attempts with two prices and two outcomes, which is both honest and billable
  • A tenant's rich message is never silently reduced without the fallback being visible in the attempt trail
  • The retryability map per channel is explicit, so a new channel's errors are classified rather than guessed
What it costs
  • Tenants must learn per-channel rules the platform could have hidden, which raises the integration cost the developer journey measures
  • Capability probes add latency and a dependency on a channel's own availability before the real send
  • Multi-channel sends need the tenant to author per channel, or to accept a plainer message everywhere
Choose differently when
If the roadmap committed to many channels for tenants who genuinely author once and send everywhere — a notification product rather than a messaging API — the thick adapter becomes right, and the editorial responsibility is one the product should take. The boundary should be re-examined the moment a tenant segment appears that never authors per channel.
Why it holds up over time
Channels will keep diverging, because their differences are commercial rather than technical. A thin boundary ages better than a translation layer, because a translation layer has to be updated every time a channel changes and is wrong in the interval; a declared capability is merely incomplete.
LessonHiding a difference is taking responsibility for it. That is sometimes right, but it should be a decision with a named owner rather than a convenience that arrives with the abstraction.
Shown on views09 15
ADR-11

Carrier protocol sessions live in a dedicated stateful tier, not in the attempt fleet

Accepted

SMPP binds are long-lived TCP sessions with sequence windows and allowlisted source addresses. Do they live inside the attempt workers, or in a tier of their own?

Context
Co-locating binds with the attempt workers removes a network hop from the hot path and keeps the deployment simpler. It also makes every autoscaling event a protocol event: a scale-in tears down a bind with messages in flight, each of which becomes an ambiguous submit — the single most expensive outcome in the platform. It makes the carrier's source-address allowlist a function of the fleet's ephemeral egress, and it couples the attempt fleet's scaling policy to how many binds a carrier permits, which is usually a small number fixed by contract. A dedicated tier costs a hop and an extra deployable, and gives each bind a stable identity, controlled termination and an egress address a carrier has approved.
Decision
Carrier protocol sessions run in a dedicated stateful tier with stable per-bind identity, individual recovery, controlled draining, and reserved egress addresses allowlisted per region. The attempt fleet calls that tier over an internal contract and scales freely. Session loss, re-bind storms and sequence-window exhaustion are treated as ordinary operating events with defined handling, not incidents.
How it is realised on the stack
A GKE StatefulSet gives each bind a stable identity and a termination hook that drains in-flight attempts before closing, so a deliberate scale-down produces no ambiguity. Egress leaves through Cloud NAT with reserved addresses, one small range per region, allowlisted by each carrier in advance — which makes a second region a correspondence exercise with each carrier, tracked as a per-route dependency with its own state. Re-binds are capped and jittered to avoid a thundering herd after a carrier restart. In-flight attempts at the moment of an uncontrolled session loss are resolved as ambiguous under ADR-05, rather than silently retried.
Options weighed
  • ChosenDedicated stateful session tier with reserved egress: Scaling the attempt fleet stops being a protocol event; costs one internal hop and a second deployable with its own scaling axis
  • RejectedBinds co-located in the attempt workers: One less hop, and every scale-in becomes a teardown with in-flight ambiguity and an egress address the carrier has not approved
  • DeferredDedicated virtual machines per carrier: Maximum isolation and stable addressing; worth revisiting for a small number of very large carriers where per-bind tuning matters
  • Right elsewhereDelegate all carrier connectivity to aggregators over HTTP: Right for a platform that never wants protocol work; forfeits direct routes, which is where margin and quality control live
Consequences
What it buys
  • The attempt fleet autoscales without creating ambiguous submits, which protects the platform's most expensive failure class
  • Each bind can be drained, restarted and tuned individually without affecting the others
  • Egress addresses are a small, stable, pre-approved set per region rather than a function of fleet size
  • Carrier-permitted bind counts stop constraining the attempt fleet's scaling policy
What it costs
  • An internal hop inside the interactive latency budget, and a second deployable with its own failure modes
  • A stateful workload in a platform that is otherwise entirely stateless, with the operational knowledge that implies
  • Regional failover depends on carriers approving a second address range on their timetable, not the platform's
Choose differently when
If the route estate moved entirely to HTTP aggregators, the session tier would have nothing to hold and should be removed rather than maintained. Conversely, if direct carrier relationships grew to dominate margin, the deferred per-carrier virtual machine option becomes attractive for the largest few.
Why it holds up over time
Stateful protocol endpoints and elastic stateless fleets have incompatible lifecycles, and that incompatibility does not improve with better orchestration. Isolating the stateful part behind an internal contract is the general answer, and it keeps the blast radius of the strange component small while the rest of the platform evolves freely.
LessonWhen one component's lifecycle fights the platform's, give it a tier rather than a workaround. The hop is cheaper than letting every scaling event become a correctness question.
Shown on views08 15 17 21

Consent, jurisdiction and evidenceThe rules the platform will be fined for, and the records that answer for them years later.

ADR-12

The platform owns regulated keywords; the tenant owns marketing preference

Accepted

Somebody texts STOP to a short code four tenants share. Whose job is it to make the messages stop?

Context
If consent belongs entirely to the tenant, the platform is a courier that cannot enforce the one rule it will be fined for, and an inbound STOP has nowhere authoritative to land: it is forwarded to whichever tenant the platform guesses, and the other three keep sending. If consent belongs entirely to the platform, it takes on liability for every tenant's marketing preferences and puts a strongly consistent store on the hot path of every submit. The split is obvious and is its own commitment: two consent systems that can disagree about the same destination, and a boundary that has to be explained to every tenant and every regulator.
Decision
The platform owns the regulated keywords and the consent evidence. An inbound STOP, UNSUBSCRIBE or locale equivalent is intercepted before the tenant is notified: the consent change is written to an append-only history, applied immediately, the mandated confirmation is sent, and the message is forwarded to the tenant as an already-actioned event. Tenants own marketing preference and may suppress more, never less. Consent is evaluated on the submit path for every message in a class that requires it, and that evaluation fails closed.
How it is realised on the stack
Inbound endpoints verify and deduplicate by route and provider reference, then the keyword interceptor writes a consent event to Spanner before anything is dispatched to the tenant. Consent events are append-only and queryable by destination, so "what did we know when we sent this" is answerable years later, and the message's refusal carries the consent record version consulted. Shared sender identities resolve tenant ownership by an explicit allocation rule rather than by conversation history. Propagation to the data plane is bounded at 30 seconds and published as a commitment, which makes messages already accepted and in flight a reported number rather than an unexamined gap.
Options weighed
  • ChosenPlatform owns regulated keywords; tenant may suppress more: The rule the platform is fined for is the rule it enforces; costs a strongly consistent read on every submit and a boundary to explain
  • RejectedTenant owns all consent; platform forwards keywords: Lower liability on paper, and a STOP on a shared number silences at most one of four tenants
  • RejectedPlatform owns all consent including marketing preference: Eliminates disagreement and makes the platform the arbiter of every tenant's marketing policy, which it is not equipped to be
  • RejectedPer-tenant choice of who owns consent: A configurable regulatory boundary is a boundary that will be configured wrongly exactly once
Consequences
What it buys
  • A STOP on a shared sender identity is honoured across every tenant using it, immediately
  • Consent evidence is append-only and queryable per destination, which is what a regulator or a complaint actually needs
  • A refusal is attributable: machine-readable reason plus the rule and consent versions consulted
  • Tenants cannot accidentally under-suppress, because the floor is the platform's
What it costs
  • A strongly consistent consent read sits inside the submit latency budget on every message in a consent-bearing class
  • Consent evaluation must hold 99.99%, making it one of the platform's availability ceilings
  • Two consent systems exist, and the boundary between them has to be documented for tenants and regulators alike
Choose differently when
A platform with strictly dedicated sender identities per tenant could leave consent with the tenant, because the shared-number problem disappears and liability follows the identity. It would also change in a jurisdiction that made the sender legally responsible in a way the platform could evidence without holding the record.
Why it holds up over time
Where liability sits is a legal fact, not an engineering preference, and the rule is simple and lasting: the party that will be fined must be the party that can enforce. Shared identities and multi-tenancy are not going away, so neither is the need for a platform-owned floor beneath tenant preference.
LessonPut enforcement where the liability is. A platform that can be fined for something it cannot prevent has drawn a boundary for its own convenience rather than for the obligation.
Shown on views06 15 21 22
ADR-13

Jurisdictional rules are versioned data that fails closed, and changes without a deployment

Accepted

Sender-ID registration requirements, quiet hours, template pre-approval and frequency caps differ across 190 jurisdictions and change without notice. Is that code or data?

Context
As code, every regulatory change is a release: a pull request, a pipeline, a deploy window, and an interval during which the platform is knowingly non-compliant in one country. As data, a change is a configuration publication that takes effect in seconds — and is therefore a path by which behaviour changes with none of the review that code gets. The second risk is the real one, and it is manage; the first is not, because the platform does not control when a regulator acts. There is a second question underneath: when the rule set cannot be read, does the platform send or refuse? Refusing blocks legitimate traffic; sending risks a fine and a suspended sender identity, which blocks all of the tenant's traffic for longer.
Decision
Jurisdictional rules are versioned control-plane data, published through a reviewed path, effective in the data plane within 30 seconds, and rolled back by moving a version pointer. Every compliance refusal records the rule-set version consulted. The data plane runs on a cached rule set with a bounded staleness ceiling, and past that ceiling compliance fails closed while everything else in the platform continues to degrade gracefully.
How it is realised on the stack
Rules live in Spanner as versioned rows and are cached in Memorystore with an explicit invalidation path and a staleness ceiling measured in seconds. The publication path is reviewed like code — a change request, a diff against the live version, a blast-radius preview naming the tenants and routes affected, and two-person approval — and publishes a new version plus a pointer move. The compliance evaluator stamps the version onto every refusal and every acceptance, so a dispute is answered with the rules as they were rather than as they are. Fail-closed is scoped deliberately: it applies to compliance only, while route quality, lease state and projections all fail open or degrade.
Options weighed
  • ChosenVersioned data, reviewed publication, fail closed on staleness: A regulator's change is live in seconds and auditable; a behaviour-changing path exists outside the code pipeline and needs equal review
  • RejectedRules as code, shipped by deployment: One review path and one audit trail, and a release cycle between a regulation changing and the platform obeying it
  • RejectedVersioned data that fails open on staleness: Keeps traffic flowing and risks a fine and a suspended sender identity, which costs the tenant far more than the refused messages
  • Right elsewhereDelegate jurisdictional compliance to each carrier: Right for a pure wholesale reseller; here the platform holds the tenant relationship and the regulator will ask it, not the carrier
Consequences
What it buys
  • A regulatory change is live in 30 seconds without a release, which is the only tempo that matches the regulator's
  • Every refusal is defensible years later, because the rule version consulted is on the record
  • Rollback is a pointer move rather than a revert and redeploy
  • Fail-closed is scoped to one evaluation, so the rest of the platform keeps its graceful degradation
What it costs
  • A behaviour-changing path exists outside the code pipeline, and it needs the same review or the gate has only been moved
  • A rule-set outage past the staleness ceiling stops sending for affected classes, which is a self-inflicted availability event
  • Blast-radius preview is extra machinery that only earns its keep on the day somebody publishes a wrong rule
Choose differently when
If the rule set stabilised — a platform in three jurisdictions with settled regulation — code would be the better home, because one review path is simpler than two. It would also change if a trustworthy third-party compliance service emerged with a liability position the platform could rely on.
Why it holds up over time
Configuration that changes behaviour needs the review that code gets; the fact that it is data does not make it safe. That is a lasting principle, and so is the asymmetry underneath the fail-closed choice: refusing some traffic is recoverable, while a suspended sender identity is not recoverable on the platform's own timetable.
LessonIf data can change behaviour, it is code with a faster deployment pipeline. Give it the same review, or admit that the gate was removed rather than relocated.
Shown on views13 18 21
ADR-14

Receipts and consent outlive the bodies they describe, and store-no-body is a supported mode

Accepted

Message content, the record of what was done with it, and the proof of permission all have retention requirements. Do they share a policy?

Context
One retention policy is simpler and wrong in both directions. Content is a privacy liability: tenants want it gone quickly, several want it never stored, and some jurisdictions require deletion. The record of what the platform did is an operational and commercial asset: a carrier invoice arrives weeks later, a dispute months later, and without the attempt and receipt history there is nothing to argue with. Consent and compliance evidence is a regulatory obligation on a different clock again, measured in years and not subject to the tenant's preferences at all. Collapsing these into one policy means either keeping content far longer than anyone wants, or destroying the evidence before it is needed.
Decision
Three clocks. Message bodies and media default to 30 days, are tenant-configurable from zero to 90, and a store-no-body mode is supported in which the platform retains metadata, attempts and receipts but never writes content. Attempts are 90 days hot and 13 months archived; receipts are 13 months regardless of body retention. Consent records, compliance verdicts and the audit log are seven years, independent of any tenant retention setting.
How it is realised on the stack
Bodies and media sit in Cloud Storage under per-tenant CMEK keys with lifecycle rules per tenant setting, physically separate from the Spanner ledger, so a purge touches no delivery record. Store-no-body is an architectural mode rather than a flag: the submit path passes content straight to the adapter and the retaining write never happens, so the promise is kept by the path not existing. Receipts land in their own partitioned store with their own lifecycle. Consent and audit go to write-once storage with Bucket Lock that the platform's own operators cannot shorten, which is what makes a seven-year claim credible.
Options weighed
  • ChosenThree clocks: content shortest, evidence longest, store-no-body supported: Each obligation gets the horizon it actually has; three lifecycle policies and three stores to reason about
  • RejectedOne retention policy across all message data: Simple, and it either keeps content too long or destroys billing and regulatory evidence before it is needed
  • RejectedTenant-configurable retention for everything including evidence: Maximally flexible, and it lets a tenant configure away the records a regulator will ask the platform for
  • Right elsewhereNever store content at all: Right for a privacy-first product; here it removes template rendering, media handling and most debugging, so it is offered as a mode rather than imposed
Consequences
What it buys
  • A body purge, including a complete store-no-body posture, leaves the delivery record and the receipts intact
  • A carrier invoice dispute arriving ten months later has evidence behind it
  • Consent and audit are credible for seven years because the storage itself prevents shortening
  • Store-no-body is a strong privacy claim made structurally rather than by configuration
What it costs
  • Three lifecycle policies across three stores, each a place to get retention wrong
  • Receipts referencing content that no longer exists are normal, and tooling must handle that gracefully
  • Store-no-body tenants lose template debugging and content-based support, which is a real product limitation
Choose differently when
A jurisdiction requiring deletion of all associated records, not just content, would force the evidence clock down and change the dispute position accordingly. If carriers moved to real-time settlement, the thirteen-month receipt horizon could shorten substantially.
Why it holds up over time
Content, the record of acting on content, and the proof of permission to act are three obligations to three different parties with three different horizons, and that will not converge. A system giving them one policy is wrong about at least two, and discovers which only when asked.
LessonRetention is not one decision. Separate the thing, the record of what was done with it, and the proof it was allowed — then let each follow the party that will ask for it.
Shown on views10 11 21

Cost and operabilityWhat a message actually costs, and how a route earns the right to carry one.

ADR-15

One rated usage event per attempt, not per message

Accepted

A message with a fallback chain may be attempted on two channels at two prices, and a long body is several segments. What is the unit of charge?

Context
Rating per message is what tenants expect and what the pricing page implies, and it cannot represent what actually happened. A fallback from a rich channel to SMS is two carrier calls with two prices. A re-route after a failure is another. A 400-character body is three segments. Rating per message forces the platform to choose one of them as canonical and lose the rest, which makes margin unknowable per route and makes a carrier invoice impossible to reconcile — the invoice is per carrier call, because that is what the carrier charged for.
Decision
A rated usage event is emitted per attempt, carrying tenant, sender identity, route, channel, traffic class, segment count and amount. The message is a grouping for the tenant's benefit, not the unit of charge. Downstream metering and invoicing decide how to present it; this platform's responsibility ends at an accurate, attributable event per carrier call.
How it is realised on the stack
The correlate-and-apply workers emit one event per attempt to a Pub/Sub topic consumed by the metering platform, with the ledger as the replayable source if the stream is lost. Events are attributed to tenant, sender identity, route, channel and class, which is what makes margin knowable per route rather than only in aggregate, and what lets the platform publish price and observed quality together. Because the ledger holds every attempt, the usage stream is reconstructible, so metering gaps are recoverable rather than permanent revenue loss.
Options weighed
  • ChosenOne rated event per attempt: Reconciles to the carrier invoice and makes per-route margin knowable; the tenant's bill needs explaining when a fallback charged twice
  • RejectedOne rated event per message, summing attempts: Matches tenant intuition, and loses the route attribution that reconciliation and margin analysis both need
  • RejectedOne event per segment: Finest grain and closest to some carrier models; multiplies event volume for information the attempt already carries
  • Right elsewhereRate inside this platform rather than emitting usage: Right where there is no separate metering platform; here it duplicates a system that exists and owns pricing
Consequences
What it buys
  • The usage stream reconciles line by line against a carrier invoice, which is what makes a dispute winnable
  • Margin is knowable per route, which is what makes least-cost routing a decision rather than a hope
  • A fallback chain's true cost is visible to the tenant and to the platform
  • The stream is reconstructible from the ledger, so a metering outage is recoverable
What it costs
  • Event volume follows attempts, not messages, so retries and fallbacks inflate it
  • A tenant seeing two charges for one message needs an explanation that the API and the console must both give
  • Attempt-level attribution has to be correct at submit time, because it cannot be reconstructed from content later
Choose differently when
If the platform ever sold a flat per-message price absorbing all attempts, the internal event grain could stay and the external presentation change — which is in fact the intended seam. Only a move to per-segment carrier billing across the whole estate would argue for a finer grain.
Why it holds up over time
The unit of charge should match the unit of cost, and the unit of cost is whatever the supplier invoices. That alignment keeps reconciliation possible and margin visible, and it survives every change of pricing model because presentation can always aggregate upward and never disaggregate downward.
LessonMeasure at the grain of the cost, present at the grain of the customer. Those are different decisions, and conflating them destroys information that cannot be recovered.
Shown on views10 14 19
ADR-16

Route eligibility is earned from measured delivery, and absence of receipts is a signal

Accepted

A carrier's contract promises a delivery rate. Should the platform route on that promise, or on what it can measure?

Context
Contracted quality is a sales document. The failure mode that matters most in this business is a route that accepts every submission and delivers nothing — no errors, no rejections, a perfect acceptance rate and a collapsed delivery rate — and it is invisible to any monitoring that watches requests rather than outcomes. A related failure is quieter still: a route that simply stops reporting receipts, which under an optimistic status model looks like improved delivery and under this platform's model looks like a rising unknown share. Both are commercial problems disguised as technical silence, and neither is discoverable without routing on measurement.
Decision
A route's eligibility and weight come from measured outcomes — delivery rate, receipt latency, unknown share and acceptance-failure rate — recomputed continuously from the receipt stream. A route with no measured outcome cannot remain eligible: the absence of receipts is treated as a signal, not as missing data. Breakers trip automatically for interactive traffic and advise for bulk, because re-routing bulk spends the tenant's money. Retirement is explicit: a route is requalified or removed, never left in the table at zero weight.
How it is realised on the stack
Dataflow aggregates the receipt stream into per-route windows in BigQuery and publishes current figures to the Memorystore quality store the selector reads. A delivery-rate collapse detector compares a route's current window against its own baseline rather than against a global threshold, so a route that has always delivered 70% is not treated like one that fell from 98%. Price and observed quality are always reported together, because least-cost routing that halves delivery rate is not a saving. The lifecycle is a loop with an explicit end state, so a dead route leaves the table rather than lingering at weight zero where it will be forgotten.
Options weighed
  • ChosenEligibility from measured outcomes, no-receipts as a signal: Catches silent collapse and silent non-reporting; quality signals lag price changes, so a cheap bad route can win briefly
  • RejectedRoute on contracted quality with manual review: Trusts a sales document and makes silent collapse a customer-reported incident
  • RejectedRoute on measured quality but treat missing receipts as neutral: Rewards carriers for not reporting, which is the easiest quality metric in the world to game
  • Right elsewhereLet tenants choose routes themselves: Right for wholesale customers with their own carrier relationships; offered as a pinned-route policy, not as the default
Consequences
What it buys
  • Silent delivery collapse becomes an automatic failover for interactive traffic rather than a support ticket
  • A carrier that stops reporting loses traffic, which turns a technical silence into a commercial conversation
  • Baselining per route avoids punishing a route that was always mediocre and rewarding one that just broke
  • Retirement is explicit, so the route table reflects reality rather than history
What it costs
  • Quality signals lag price changes, so a least-cost policy can pick a cheap bad route faster than the loop discovers it
  • A new route has no measurements, so onboarding needs a probationary period rather than immediate eligibility
  • Automatic failover spends more money without asking, which has to be bounded and reported
Choose differently when
If carriers provided trustworthy, audited, real-time quality telemetry, the platform could route on their figures and keep its own measurement as verification. Conversely, if the estate shrank to a handful of directly contracted routes, manual review with measurement as evidence would be sufficient and cheaper.
Why it holds up over time
Supplier quality should be measured by the buyer, because the supplier's incentive to report is weakest exactly when the report matters most. Treating silence as information, rather than as absence of information, is the part that most often gets lost, and it is the part that catches the worst failures.
LessonNever let a metric improve when a supplier stops reporting. If silence can look like success, the measurement is pointed at the wrong thing.
Shown on views19 20

Every package used, in one table

Twelve terms used precisely throughout the package, several of which mean something narrower here than in ordinary usage.

PackageWhat it isWhat it does hereConsidered instead
Accepted The state of a message whose submission has been validated, screened and durably appended to the ledger, and acknowledged to the caller. The platform's promise of ownership, and the only thing the submit path ever promises. Often read by tenants as "sent", which it deliberately is not — hence the separate vocabulary in the API.
Attempt One carrier call for one message, with its own route, segment breakdown, provider reference, outcome and price. The unit of retry, of routing, of rating and of the fold that produces delivery state. Sometimes called a delivery or a send, both of which imply an outcome the attempt does not carry.
Receipt An asynchronous statement from a carrier about an attempt, arriving seconds to hours later, possibly duplicated, out of order, or for an unknown reference. Evidence. It is appended and retained for thirteen months, longer than the body it describes. A delivery report or DLR in carrier protocols, which overstates how definitive it is.
Unknown The terminal state of a message whose time-to-live elapsed with no conclusive receipt. A first-class answer in the API, the SLOs and the route-quality signal — never collapsed into delivered or failed. Usually reported elsewhere as delivered, which is the lie this platform is designed not to tell.
Ambiguous submit An attempt that timed out with no response on a route offering no idempotency key, so neither delivery nor non-delivery can be inferred. A recorded fact resolved by per-route policy, with the policy applied written onto the message. Commonly treated as a plain timeout and retried, which is how duplicate login codes are made.
Route A tuple of channel, destination jurisdiction, provider and sender-identity type, carrying price per segment, a contracted throughput window, supported features, observed quality and health. The unit of pacing, failover, pricing, policy and commercial relationship. Loosely called a carrier or a connection, which conflates the provider with the specific lane bought from them.
Lease A block of permits checked out from a route's contracted throughput window and spent locally without further coordination. How a contractual window is held across an elastic fleet with no coordination hop per message. A rate limit, which understates the penalty: breaching a window suspends a sender identity.
Sender identity The thing a recipient sees as the origin: a long number, a short code, an alphanumeric sender ID, a toll-free number or a channel business identity, with the jurisdictions it is registered in. The asset a throughput breach or a compliance failure actually damages, and the key by which inbound traffic finds its tenant. Often called the "from" number, which hides that it is a registered, suspendable entity rather than a field.
Traffic class One of interactive, conversational or bulk, declared at submission. Decides the lane, the route preference, the SLO and the position in the shedding order. A priority, which implies a single ordering rather than genuinely different products.
Client reference A tenant-supplied identifier, unique per tenant, carried on the submission. The idempotency key that makes a resend return the original message, and the only identifier a tenant's support desk has. An external id or correlation id; here it has enforced uniqueness and a dedup window, which those usually do not.
Regulated keyword An inbound message such as STOP, UNSUBSCRIBE or HELP, or a locale equivalent, whose handling is mandated rather than optional. Intercepted by the platform before the tenant is told, with the consent change written first and a mandated reply sent. Treated elsewhere as tenant application logic, which is how a STOP on a shared number ends up unenforced.
Projection A rebuildable read model derived by folding a message's attempts and receipts, served for status queries. The thing tenants read. It may lag, must be rebuildable in hours, and is never the authority. A status table, which looks the same and is authoritative, which is the difference this package is about.
Open svg/<view>.svg or drawio/<view>.drawio in draw.io Desktop or at app.diagrams.net to edit. The SVG carries the diagram inside it, so it is both the picture and the source. This folder is self-contained — copy it whole and every link still resolves.