# Architecture Decision Record

*Payment Checkout & Ledger Service · Solution Architecture v1.0 · open source, self-hosted on Kubernetes across two in-country data centres · Payments Platform Architecture · 2026-10 · 24 views · 18 architecture decision records*

The argument these decisions serve is summarised in the [Architecture One-Pager](architecture-one-pager).

Eighteen decisions make up this architecture. Everything else across the twenty-four views is either a consequence of one of them or a detail that could be decided differently next quarter without anyone redrawing the set. Each record carries more than the classic context / decision / consequences triple: the forcing question, how the decision is realised on a self-managed open-source stack, the alternatives including the ones that are right for a different organisation, the conditions that would flip the choice, why it should still hold as scale, staff and technology change, and the transferable lesson.

> **Status of this document.** This is a design, not a report on a running system. Every rate, latency, interval, limit and retention figure is a stated assumption chosen to be defensible and arguable rather than measured. The operating context assumed throughout is a single-country payment service for 50,000 merchants, 12 million payment attempts a day at peak season, 420 attempts a second at steady state with a 4,000-a-second sale peak sustained for ten minutes, roughly eight ledger entries per payment and therefore about 35 billion entries a year, 30 provider settlement files a day of up to 10 million lines each, and a double-charge budget of one per million attempts.

## How to read a record

- **Question:** The forcing question: why a decision was needed at all.
- **Context:** The requirement, the scale and the constraint that make it hard.
- **Decision:** What this architecture does, stated so it can be checked.
- **How it is realised on this stack:** The concrete mechanism: which component, configured how, in which plane.
- **Options weighed:** Chosen, rejected, deferred, or right elsewhere, with the reason for each.
- **Consequences:** What the choice buys and what it costs, both kept visible.
- **Choose differently when:** The conditions that would flip the decision for your system.
- **Why it holds up over time:** What keeps the decision right as scale, staff and technology change.
- **Lesson:** The principle that transfers beyond this platform.

## Decision map

**The ledger and the record**: Where truth about money lives, what shape it takes, and what makes a posting happen exactly once.

- ADR-01 · Authorization plane and ledger plane are separate, coupled only by a durable event log
- ADR-02 · The ledger is append-only double entry, balanced at write time, and is the sole system of record
- ADR-03 · Idempotency is declared separately at every boundary, with a different key at each

**Resolving the network**: What to do when the rail will not say what happened, and whose word the platform ultimately takes.

- ADR-04 · `unknown` is a first-class outcome, and no retry ever follows one
- ADR-05 · The provider's synchronous response is a hint; the settlement file is the evidence
- ADR-06 · Routing is a versioned declarative policy whose decisions are reproducible from stored inputs
- ADR-07 · Unmatched settlement value is posted to a named, aged break account and never netted away

**Shape, scale and contracts**: How state is divided, what is promised to merchants, and how a system holding money is changed.

- ADR-08 · Payment state is sharded by merchant; the ledger gives up a global sequence
- ADR-09 · Merchant webhooks are at-least-once with a per-object sequence number and a queue per merchant
- ADR-10 · Posting rules are versioned artefacts, shipped shadowed, and gated on a replay that must be byte-identical

**Trust and key custody**: What is inside PCI scope, who may move money, and how large a leaked credential's blast radius is.

- ADR-11 · Keep the cardholder data environment small: browser to CDE directly, tokens everywhere else
- ADR-15 · Publishable and secret credentials are separate, and a leaked secret reaches only its own merchant's account

**Money, licensing and correction**: Whether the platform holds funds, who may correct the books, and when money is held rather than refused.

- ADR-12 · The settlement matcher may resolve payment state, under declared constraint
- ADR-13 · The platform holds merchant balances rather than passing funds through
- ADR-14 · Dual control is a service in the request path, not a policy in a document
- ADR-17 · A sanctions match holds a payout rather than rejecting it, and payouts move only proven cash

**Platform and operations**: Where it runs, what is derived rather than stored, and the cost of the localisation choice.

- ADR-16 · Every derived store is rebuildable, and the rebuild is run on a schedule to prove it
- ADR-18 · Self-managed open source in two in-country data centres, not a hyperscaler

## Technology by capability

The requirement document is vendor-neutral by design — "durable ordered log", not a product name — so the stack below is this design's answer rather than the requirement's. Every row names what else would be reasonable, because the decision that matters is the capability and not the component.

| Capability | Choice | Origin | Credible alternative | Why this one | Record |
|---|---|---|---|---|---|
| API edge and service mesh | Envoy behind an anycast VIP, mTLS to all backends, per-merchant concurrency and fair queuing | open source, self-hosted | NGINX or HAProxy at the edge; a managed API gateway where one is permitted | Per-merchant concurrency limiting, not just rate limiting, is the mechanism that contains a single merchant's sale peak | ADR-08 |
| Idempotency key store | Redis with 24-hour TTL, storing a request-body hash beside the resulting payment identifier | open source, self-hosted | A relational table with a TTL sweep, at lower volume | A hot in-memory lookup on the critical path, where a conflicting key reuse must be detected rather than silently served | ADR-03 |
| Durable event log | Kafka, 5 brokers stretched across both data centres, RF 3, min.insync.replicas 2, partitioned by payment identifier, 7-year retention | open source, self-hosted | Redpanda; Pulsar; NATS JetStream at lower volume | Per-payment ordering plus long retention makes the log evidence as well as transport, which is what lets any projection be rebuilt by replay | ADR-01 |
| Ledger store | PostgreSQL, append-only tables with UPDATE and DELETE revoked, a deferred zero-sum constraint per transaction, synchronous replication to DC-2, partitioned by period and account class | open source, self-hosted | A managed ledger service where permitted; CockroachDB or YugabyteDB for multi-region; TigerBeetle for pure ledger throughput | A balance check enforced by the store catches a class of bug no amount of application test coverage reliably catches | ADR-02 |
| Payment state store | PostgreSQL shards keyed by a hash of merchant identifier, shard map in the config registry | open source, self-hosted | Vitess or Citus for managed sharding; a single large instance below a few hundred attempts a second | The merchant is the isolation boundary, and every payment-path operation must stay inside one shard so no synchronous cross-shard transaction exists | ADR-08 |
| Token vault and card data proxy | Self-hosted tokenisation service on a dedicated Kubernetes cluster in its own network segment, format-preserving tokens, no PAN at rest outside it | open source, self-hosted | A PCI-compliant vault as a service; network tokens from the card schemes where coverage allows | Scope reduction beats scope hardening: the application estate only ever sees tokens | ADR-11 |
| Key custody | Network HSM pair per data centre, FIPS 140-3, no key exportable in plaintext, used for tokenisation, provider authentication and dual-control signing | vendor hardware | A cloud HSM service where permitted; SoftHSM for non-production only | An operator with full database access still cannot decrypt anything, and a dual-control approval is cryptographic evidence rather than a row | ADR-14 |
| Routing policy registry | Content-addressed YAML artefacts promoted through the release pipeline with commercial sign-off; active version stamped on every attempt | open source, self-hosted | A commercial decision-engine; a feature-flag platform for simpler weighting | A routing decision must be reproducible from stored inputs years later, which is two columns and a registry rather than a model | ADR-06 |
| Workforce identity | Self-hosted OIDC provider with phishing-resistant MFA, feeding the ops console and the dual-control service | open source, self-hosted | A managed workforce IdP where permitted | A payments back office is a credential-phishing target by definition, and the second approver's identity must be independently trustworthy | ADR-14 |
| Settlement file transport | SFTP ingest with file hashing and a run identifier, idempotent and re-runnable, restatements posting corrections | open source, self-hosted | Provider streaming APIs where offered, which most do not | The file is the evidence, so ingest must be repeatable without changing the books | ADR-05 |
| Observability | Prometheus and Grafana for metrics, OpenTelemetry collection, Loki for logs with 400-day retention, correctness alarms separated from latency and saturation | open source, self-hosted | A managed observability platform where egress of operational metadata is permitted | Only the correctness row pages: an unbalanced-transaction rejection or a bank-versus-ledger delta, not a latency percentile | ADR-16 |
| Compute platform | Self-managed Kubernetes, three control-plane nodes per data centre, separate cluster for the CDE | open source, self-hosted | A managed Kubernetes service in an in-country region where the regulator accepts the provider | Residency, PCI scope containment and dependency approval drove it, not a preference for owned hardware | ADR-18 |

## The decisions, and the alternatives that lost

### The ledger and the record

*Where truth about money lives, what shape it takes, and what makes a posting happen exactly once.*

#### ADR-01 · Authorization plane and ledger plane are separate, coupled only by a durable event log

**Status:** Accepted  ·  **Shown on views:** 07, 13, 24

*When a payment succeeds, does the book of record move inside the buyer's request, or after it?*

**Context.** A payment service has two jobs with incompatible failure profiles. Answering a buyer needs single-digit-hundreds of milliseconds of platform time, tolerates declining, and is worth failing rather than hanging. Keeping the books needs strong consistency, synchronous replication and an append-only store that may never lose or double an entry, and is worth waiting for. Putting both inside one request means the slowest, strictest store in the system sits on the buyer's critical path, so a ledger partition during a sale becomes a checkout outage at 4,000 attempts a second. Separating them means the platform will, for a window, have told a buyer and a merchant that a payment succeeded while the ledger says nothing about it — and that window never fully closes on its own.

**Decision.** Split the architecture at that boundary. The authorization plane is stateless, deadline-bound, holds no truth and is permitted to decline, time out and fail closed. The ledger plane is append-only, strongly consistent, synchronously replicated, and permitted to be unavailable for writes for minutes. The only coupling between them is a durable ordered event log; no service in the authorization plane writes the ledger, and no ledger operation sits on a buyer request. Accept the resulting window explicitly, make its width a first-class metric, and build the resolution machinery that guarantees it always closes.

**How it is realised on this stack.** Intent and routing services run as stateless pods on self-managed Kubernetes across both in-country data centres, scaled on concurrent in-flight provider calls rather than request rate. Every state transition is appended to Kafka, partitioned by payment identifier so a single intent's events are strictly ordered, with seven-year retention so the log is evidence rather than a buffer. A dedicated ledger poster is the only consumer that writes PostgreSQL's append-only ledger tables, which replicate synchronously to the second site at 1.4 ms RTT. Posting lag p99 and the age of the oldest unposted event are dashboard metrics with alarms, not debug counters.

| Option | Verdict | Reasoning |
|---|---|---|
| Separate planes coupled by a durable ordered log | Chosen | Checkout survives a ledger partition; the books stay strict; the cost is a visible, bounded, always-closing window |
| Synchronous ledger write inside the buyer's request | Right elsewhere | Right for a low-volume service where a ledger write is a few milliseconds and no sale peak exists — and it removes an entire class of reconciliation work; wrong at 4,000 attempts a second, where it converts every store hiccup into lost revenue |
| One service owning both state machine and ledger, posting asynchronously inside itself | Rejected | Keeps the window but loses the structural guarantee: with both in one process, the first urgent incident produces a direct ledger write and the boundary is gone |
| No ledger; derive balances from the payment-state table on read | Rejected | Cheapest to build and impossible to audit: there is no answer to what the platform believed last Tuesday, and corrections become edits |

**What it buys**

- A ledger store partition degrades to a lagging balance rather than a failed checkout, which at a sale peak is the difference between a slow dashboard and lost revenue
- The ledger can be as strict as it likes — synchronous replication, balance checks at write, no caching — because nothing in it is latency-critical
- The event log is evidence as well as transport: any projection, including the ledger itself, can be rebuilt by replay
- Reporting, analytics and settlement workloads read stores that are nowhere near the payment path

**What it costs**

- A permanent window exists in which a payment has succeeded in the world and is not in the books, and in the `unknown` case it can be 24 hours wide
- Four components — the poller, the callback resolver, the settlement matcher and the break manager — exist only to close that window and are pure overhead when the rails behave
- A merchant can be told 'paid' before their balance moves, which generates support contacts that a synchronous design would not
- Two durability stories to operate rather than one: a broker and a relational store, both self-managed

**Choose differently when.** If volume were an order of magnitude lower and there were no sale peak, a synchronous ledger write inside the request would be simpler, would remove the window entirely, and would delete four components. The decision rests on three facts: a 4,000-a-second peak confined to minutes, a ledger that must be synchronously replicated for RPO 0, and a double-charge budget that makes optimistic retries unavailable. Remove the peak and this record is the wrong one.

**Why it holds up over time.** The split is a statement about where truth lives, not about Kafka or PostgreSQL. Both can be replaced — and will be — without changing the sentence "the buyer-facing path may fail; the book of record may not be wrong." Every later view in this set either honours that sentence or contradicts it.

> **Lesson.** When two halves of a system have incompatible failure profiles, the right move is usually to separate them and pay for the gap honestly, rather than to force the stricter half onto the faster half's latency budget. The cost of the separation is a window — and a window you have named, measured and built machinery to close is a far smaller liability than a shared path you have quietly made fragile.

#### ADR-02 · The ledger is append-only double entry, balanced at write time, and is the sole system of record

**Status:** Accepted  ·  **Shown on views:** 11, 12, 20

*What is the authoritative representation of money in this platform — a balance, or a history?*

**Context.** The tempting design is a payments table with a status column and a merchant balance beside it, updated in the same transaction. It is simple, fast, and gives the wrong answer to the only questions that ultimately matter: what did the platform believe about this payment last March, why does this merchant's balance differ from the sum of their payments by 412 rupees, and who changed it. Double-entry bookkeeping answers all three by construction, at the price of roughly eight rows per payment instead of one, an account model that has to be designed before the first payment, and the loss of the ability to simply fix a number.

**Decision.** Record every money movement as a transaction of two or more entries whose signed amounts sum to zero, against named accounts. Reject an unbalanced transaction at write time rather than correcting it later. Make the ledger append-only — no update, no delete, and no privileged path that bypasses the balance check. Derive every entry from a declared, versioned posting rule keyed on event type. Store amounts as integer minor units with an explicit currency. Every balance, statement and report is a projection of this ledger, and a number that cannot be derived from it is a number the platform does not publish.

**How it is realised on this stack.** PostgreSQL with `ledger_transaction` and `ledger_entry` tables, a deferred constraint asserting that the signed `amount_minor` of a transaction's entries sums to zero, and `REVOKE UPDATE, DELETE` on both tables for every role including the application's. The named account tree — merchant payable, platform fee revenue, provider receivable, tax withheld, dispute reserve, settlement clearing, break — is reference data with its own history. Posting rules are versioned artefacts in a registry, and `posting_rule_version` is stored on every transaction. Closed fiscal periods are archived at partition level with query support preserved.

| Option | Verdict | Reasoning |
|---|---|---|
| Append-only double entry, balanced at write, as sole record | Chosen | Auditable by construction; corrections are new entries; every report is derivable |
| Payment table with a status column and a cached balance | Right elsewhere | Right for a pre-revenue product or an internal tool where nobody will ever ask what you believed last quarter; wrong the moment a regulator, an auditor or a disputing merchant does |
| Single-entry transaction log with balances derived by summing | Rejected | Derivable but not self-checking: nothing catches a posting that credits without a corresponding debit, which is exactly the bug class double entry exists to catch |
| An off-the-shelf ledger service or blockchain-backed ledger | Rejected | A managed ledger would be reasonable in a cloud this design cannot use (ADR-18); a blockchain adds an immutability guarantee the platform already has and a consensus cost it does not need |

**What it buys**

- An unbalanced transaction is impossible rather than unlikely: the store refuses it, so the bug surfaces as a rejection in the poster rather than as a wrong number in a report
- Any question about history is answerable from the ledger plus the posting-rule version, including years later
- Corrections are new opposite-signed entries, so there is no mechanism by which history changes and therefore no need to trust that nobody used one
- A daily trial balance is a real check rather than a report, because an imbalance means a bug and not a rounding issue

**What it costs**

- Roughly eight entries per payment — about 35 billion entries a year and 12 TB before compression — which makes partitioning and archival a day-one concern rather than a later one
- The account model must be designed before the first payment, and getting it wrong is expensive once billions of entries reference it
- Nobody can fix a number. Every correction is a workflow, which is correct and is also slower than an operator would like
- Integer minor units mean every interface and every report must agree about currency exponents, and a single float anywhere in the chain is a defect

**Choose differently when.** If the platform were pure pass-through and never held a balance (see ADR-13), the account model would shrink dramatically and a simpler transaction log might carry it. And if there were no regulatory retention requirement and no disputes, the self-checking property would be worth less than the storage it costs. Neither condition holds here: the platform is custodian of merchant funds and must reconstruct any payment for ten years.

**Why it holds up over time.** Double-entry bookkeeping predates every technology in this set by centuries and will outlive all of them. Choosing it is choosing not to invent a money model — the one part of this architecture that is certain not to need revisiting, however many times the store beneath it is replaced.

> **Lesson.** Choose a representation that makes the errors you fear structurally impossible rather than merely unlikely. A balance check enforced by the store catches a class of bug that no amount of test coverage over application code reliably catches, because it holds for code nobody has written yet.

#### ADR-03 · Idempotency is declared separately at every boundary, with a different key at each

**Status:** Accepted  ·  **Shown on views:** 12, 13, 14

*Which duplicate is being prevented — the buyer's second tap, the provider's repeated callback, or the poster's replayed event?*

**Context.** "The system is idempotent" is the sentence that hides the bug. There are at least four distinct duplication events in a payment flow and they need four different keys. A buyer double-tapping produces two HTTP requests carrying the same merchant-supplied `Idempotency-Key`. A provider retrying a webhook produces two callbacks carrying the same provider event identifier. A consumer restart replays events, producing two attempts to post the same ledger transaction. A payout retry produces two instructions to a bank. An edge-only key protects against the first and none of the others; a ledger-only key protects the books and lets duplicate provider calls happen, which is a double charge.

**Decision.** Declare the natural key at each boundary explicitly and enforce it there. At the edge: the merchant's `Idempotency-Key`, scoped per merchant, where a repeat with an identical body returns the original intent and a repeat with a differing body is a conflict. At the callback: the provider's own event identifier. At the ledger: the payment event's identifier as a unique constraint on the ledger transaction, which is what makes posting exactly-once without coordination. At the payout: a unique instruction key derived from merchant, period and amount. Four keys, four guarantees, documented separately.

**How it is realised on this stack.** Redis holds edge idempotency keys with a 24-hour TTL, storing a hash of the request body alongside the resulting payment identifier so a conflicting repeat can be detected rather than silently served. Callback dedup is a unique index on `(provider_id, provider_event_id)` in the payment shard. The ledger carries `UNIQUE (event_id)` on `ledger_transaction`, so a replayed event's insert fails on the constraint and the poster treats that failure as success. The payout table carries `UNIQUE (instruction_key)`.

| Option | Verdict | Reasoning |
|---|---|---|
| A declared natural key at each of the four boundaries | Chosen | Each duplication event has an owner; replay safety at the ledger needs no distributed coordination |
| One idempotency key threaded through the whole flow | Rejected | The merchant's key does not exist at the callback boundary and is not unique per provider event; threading it requires inventing a mapping that is itself a store with its own races |
| Exactly-once processing via a distributed transaction across broker and ledger | Rejected | Buys what a unique constraint already buys, at the cost of a two-phase commit on the ledger's write path |
| Deduplication windows with time-bounded caches at each hop | Right elsewhere | Right where no stable natural key exists — some legacy rails genuinely have none; wrong here, because a window that expires is a correctness bug waiting for a slow retry |

**What it buys**

- The ledger poster can be replayed arbitrarily: a replay of the entire log produces a byte-identical ledger, which is the release gate in ADR-10
- A buyer's double tap is answered with the original intent and produces no second provider call, which is where double charges actually come from
- Each guarantee is independently testable, so a property test can target one boundary at a time
- A conflicting idempotency-key reuse is reported as a conflict rather than quietly creating a second payment — a merchant integration bug surfaces as an error

**What it costs**

- Four mechanisms to document, test and explain, and a reviewer must understand which one covers which failure
- The edge key store is a hot in-memory dependency on the critical path; its loss degrades duplicate protection even though the ledger stays safe
- The 24-hour edge TTL is a judgement call: a retry at 25 hours creates a second intent, and no window is obviously correct
- Treating a unique-constraint violation as success is correct and reads like a bug to anyone encountering it for the first time

**Choose differently when.** If every provider offered a reliable idempotency key on authorization requests, the edge key and the no-retry-after-unknown rule in ADR-04 could both relax, because a retry would be provably safe. Most rails do not, and the ones that do disagree about semantics. If that changed industry-wide, this record and ADR-04 would both be rewritten.

**Why it holds up over time.** The keys are properties of the boundaries, not of the technology at them. A different broker, store or cache changes where the constraint is declared and not which key it is on — and a unique constraint on a natural key will remain the cheapest exactly-once mechanism available for as long as databases have unique constraints.

> **Lesson.** "Idempotent" is not a property of a system; it is a property of an operation with respect to a specific key. Naming the key at every boundary turns a vague reassurance into four testable guarantees, and usually reveals at least one boundary nobody had considered.

### Resolving the network

*What to do when the rail will not say what happened, and whose word the platform ultimately takes.*

#### ADR-04 · `unknown` is a first-class outcome, and no retry ever follows one

**Status:** Accepted  ·  **Shown on views:** 04, 14, 24

*The provider did not answer within the deadline. Is the buyer charged?*

**Context.** An authorization that times out is not a failure. The issuer may have approved it, the funds may be held, and the buyer's statement may already show it — the platform simply does not know. Treating it as a failure and retrying is the single most reliable way to charge someone twice. Treating it as a success is worse. The honest third state is uncomfortable for everyone: the buyer sees a spinner resolve into 'processing', the merchant cannot decide whether to ship, and the platform has an open obligation it must track to closure. There are three ways out — void-and-retry, poll-until-known, and defer to the settlement file — and they differ by hours and by risk.

**Decision.** Normalise every provider response into one of four results — `approved`, `soft_decline`, `hard_decline`, `unknown` — and treat `unknown` as a distinct outcome with its own resolution path, never as an error. An intent whose attempt returned `unknown` moves to `processing`, a terminal-pending state that only an inbound provider signal or a reconciliation job may leave. No second provider attempt is ever made after an `unknown`, by any path, including a buyer retry or an operator action. The resolution ladder is provider inquiry where supported, then the daily settlement file. Where a choice exists between declining a legitimate payment and risking a double charge, decline.

**How it is realised on this stack.** The adapter enforces an 8-second hard deadline and maps timeout, connection reset and any ambiguous provider code to `unknown`, keeping the raw code alongside. The intent state machine declares no legal transition from `processing` whose source is a client request, so a retry is structurally impossible rather than merely discouraged. The outcome poller picks up intents in `processing` beyond 60 seconds and issues provider inquiries with backoff where `provider.supports_inquiry` is true. Unresolved intents are picked up by the settlement matcher (ADR-12). The buyer-facing response for `processing` says explicitly not to retry, and the merchant webhook carries `processing` rather than silence.

| Option | Verdict | Reasoning |
|---|---|---|
| Hold in `processing`; inquire, then defer to the settlement file; never retry | Chosen | Cannot double-charge; resolves in seconds when inquiry is supported and up to 24 hours when it is not |
| Void the ambiguous transaction, then retry | Right elsewhere | Right where every provider reliably honours a void for a transaction it has not acknowledged, which some single-acquirer setups genuinely have; wrong across four providers with differing void semantics, because a void that silently no-ops turns the retry into a double charge |
| Retry against a different provider | Rejected | The worst option available: the second provider cannot see the first's hold, so a success on both is undetectable until settlement |
| Treat the timeout as a decline and let the buyer try again | Rejected | Simplest and the classic cause of duplicate charges; it moves the retry decision to the buyer, who has no information |

**What it buys**

- The double-charge budget of one per million attempts is achievable, because the action that causes double charges is structurally unavailable
- `processing` is an honest product state the merchant can reason about, rather than a lie in either direction
- Buyer retries are free: the edge idempotency key returns the same intent and no new attempt is made
- The resolution path is the same machinery as ordinary reconciliation, so it is exercised daily rather than only during incidents

**What it costs**

- Resolution can take up to 24 hours where the provider supports no inquiry, which is a genuinely poor buyer experience
- Some merchants will ship on `processing` despite being told not to, and some of those payments will resolve as declined
- A provider outage produces a growing population of `processing` intents, which is operationally alarming even when it is working as designed
- Conversion is lost: a buyer told 'processing' sometimes abandons and pays by another method elsewhere

**Choose differently when.** If every provider in the routing set supported a reliable inquiry API answering within seconds, the 24-hour tail would disappear and `processing` would become a sub-second state — at which point void-and-retry becomes unnecessary rather than merely risky. Conversely, if the business decided that a double charge refunded within the hour was cheaper than lost conversion, the retry ban would be the wrong rule, and this record would be the first to go.

**Why it holds up over time.** New rails arrive constantly and every one of them will occasionally fail to say what happened. A design that treats ambiguity as a first-class state absorbs a new rail as an adapter; one that treats it as an error rewrites itself for each. The rule survives every change of provider, protocol and decade.

> **Lesson.** When an external system can leave you genuinely uncertain, make the uncertainty a named state in your model rather than collapsing it into success or failure. The collapse is always where the expensive bug lives, and it is always the thing a first design does.

#### ADR-05 · The provider's synchronous response is a hint; the settlement file is the evidence

**Status:** Accepted  ·  **Shown on views:** 09, 16, 20

*When the provider's API and the provider's settlement file disagree, which one does the platform believe?*

**Context.** A provider speaks to the platform through at least three channels that routinely disagree: a synchronous API response, asynchronous callbacks, and a daily settlement file. The API response is fastest and least reliable — it can time out after approval, report success for a transaction later reversed, or carry a code whose meaning differs between providers. Callbacks arrive duplicated, out of order, hours late, or occasionally for transactions the platform has no record of. The settlement file is slow, sometimes restated, and is the thing the provider will actually pay against. A platform that treats the API response as truth books revenue it may not receive; one that waits for the file cannot answer a buyer.

**Decision.** Rank the channels explicitly. The synchronous response advances the payment state machine and may be reported to the buyer and merchant. The callback resolves ambiguity and may advance state, subject to deduplication and a legal-transition check. The settlement file is the evidence: it is what reconciliation matches against, what payouts are computed from, and what wins any disagreement. Persist every raw callback payload before interpreting it, so a parser bug destroys no evidence, and ignore — while recording — any callback whose implied transition is illegal from the current state.

**How it is realised on this stack.** The callback endpoint verifies signature and freshness (5-minute window, nonce replay rejected) at the perimeter, writes the raw body to a 7-year payload store keyed by a content hash, and only then parses. Deduplication is a unique index on `(provider_id, provider_event_id)`. The state resolver consults the intent's declared legal transitions and records an `ignored_transition` event rather than dropping a callback silently. Settlement files land on SFTP, are hashed and recorded with a run identifier, and matched on `provider_ref` plus amount against `payment_attempt`.

| Option | Verdict | Reasoning |
|---|---|---|
| Ranked channels: response advances, callback resolves, file decides | Chosen | Fast enough for a buyer, authoritative where money is concerned, and the ranking is explicit rather than emergent |
| Treat the synchronous response as truth and reconcile as a back-office exercise | Right elsewhere | Right for a single-acquirer, low-volume integration with a reliable partner and small amounts; wrong where settlement disagreement is routine and the platform pays merchants before it is paid |
| Wait for the settlement file before reporting any outcome | Rejected | Correct and unusable: no buyer waits 24 hours, and no merchant can run a storefront on T+1 outcomes |
| Trust whichever channel arrived most recently | Rejected | Not a ranking at all; with out-of-order callbacks it makes the final state depend on network timing |

**What it buys**

- Revenue is only ever recognised against evidence the provider will pay, so the books do not drift optimistically
- A parser bug or a schema change is recoverable: the raw payloads are kept and can be reprocessed
- Out-of-order and duplicate callbacks are one mechanism — dedup plus legality — rather than four separate races
- An ignored callback is recorded, so 'we never got told' is an answerable question

**What it costs**

- Three channels to integrate, monitor and version per provider, each with its own failure modes
- Raw payload storage for seven years is a meaningful volume and a data-residency obligation in its own right
- Buyer-visible state can be contradicted later by the file, which is honest and generates support work
- A restated file means yesterday's reconciliation output can change, so no reconciliation result is ever final until the period closes

**Choose differently when.** If the platform settled directly on a rail it operated, there would be no external file to reconcile against and the API response would be the truth by definition. That is the position of a network rather than a payment service, and it is explicitly out of scope. Within scope, the ranking would only change if a provider offered a synchronous response with a contractual settlement guarantee attached — which none does.

**Why it holds up over time.** The ranking is about trust, not technology. Protocols will change from SFTP files to streaming APIs and the question — which channel will the counterparty actually pay against — will have the same answer and the same ranking.

> **Lesson.** Where a counterparty speaks through several channels, rank them by what they commit to rather than by how fast they are, and store the raw form of each before interpreting it. The slowest channel is usually the one with money behind it, and the one you will wish you had kept verbatim.

#### ADR-06 · Routing is a versioned declarative policy whose decisions are reproducible from stored inputs

**Status:** Accepted  ·  **Shown on views:** 02, 08, 13

*Why did this payment go to this acquirer, and can that be answered in three years?*

**Context.** With four providers across two rails, every attempt needs a routing decision, and that decision has commercial, reliability and regulatory consequences. The options run from static per-method configuration (auditable and dumb) through a rules engine evaluated per attempt (expressive, and a second production system with its own correctness problem) to an outcome-learning optimiser (highest value, and nearly impossible to explain to an auditor asking why this payment went where it did). Underneath sits a harder requirement: a disputing merchant or a regulator may ask about a specific payment years later, and "the model decided" is not an answer.

**Decision.** Express routing as a declarative policy evaluating payment method, issuer or bank identifier, amount band, merchant preference, provider health and provider cost. Version the policy as an immutable artefact and store `policy_version` on every attempt, so any decision is reproducible from stored inputs. Make cost an input that is strictly subordinate to approval rate and to any merchant-declared constraint. Derive provider health from observed approval rate and latency, never from a provider's status page. Allow providers to be added, weighted, drained and removed by configuration change, without deploying the authorization path. Defer outcome-learning routing to Phase 3, and only with reproducible decision records.

**How it is realised on this stack.** Policies are YAML artefacts in a registry, content-addressed and promoted through the same pipeline as code with a required commercial sign-off. Pods hold the active policy in memory and reload on registry change; the active version identifier is stamped on every `payment_attempt` row alongside the raw provider code. Health is computed from a rolling window of observed outcomes per provider and per method, feeding both the circuit breaker and the policy's health input. Draining a provider is a weight change, not a deployment.

| Option | Verdict | Reasoning |
|---|---|---|
| Versioned declarative policy, decisions reproducible from stored inputs | Chosen | Expressive enough for real routing, auditable years later, changeable without a deployment |
| Static per-method configuration | Right elsewhere | Right with one or two providers and no commercial routing requirement, where the simplicity is worth more than the expressiveness; wrong with four providers and per-issuer approval differences worth real money |
| Outcome-learning cost-and-approval optimiser | Deferred | The highest-value option and the hardest to explain; revisit in Phase 3 once decision records are reproducible, which is what this record puts in place |
| Let each merchant specify the provider | Rejected | Hands routing risk to people with no visibility of provider health, and makes draining a failing provider a customer-communication exercise |

**What it buys**

- Any routing decision is reconstructable from the stored policy version and the attempt row, which is what makes a regulator's question answerable
- Draining a degraded provider is a configuration change evaluated on the next attempt, not a release
- Health derived from observed behaviour reacts faster than a provider's own status reporting and is not subject to its incentives
- Cost being structurally subordinate to approval rate prevents the common failure where routing quietly optimises margin into lost conversion

**What it costs**

- A policy language is a second production system with its own validation, testing and review needs
- Immutable policy versions accumulate, and reproducing a 2027 decision in 2030 means keeping the evaluator able to run a 2027 policy
- Declarative routing leaves value on the table compared with a learned router, and that gap grows with provider count
- A policy change with a sign-off gate is slower than a commercial team would like, which creates pressure to add overrides

**Choose differently when.** If the platform ever ran with a single acquirer, the whole policy mechanism would be dead weight and static configuration would be correct. In the other direction, once the provider set grows past roughly ten and per-issuer approval differences dominate, declarative routing stops paying and the learned router in Phase 3 becomes the right answer — carrying this record's reproducibility requirement with it.

**Why it holds up over time.** The durable part is not the policy language but the requirement that a decision is reproducible from stored inputs. That requirement survives the move to a learned router, which is exactly why it is written down before the router exists rather than after.

> **Lesson.** When a decision has commercial consequences and a long audit tail, store the inputs and the decision-maker's version alongside the outcome. It costs two columns and it is the difference between answering a question about 2027 and reconstructing it from code archaeology.

#### ADR-07 · Unmatched settlement value is posted to a named, aged break account and never netted away

**Status:** Accepted  ·  **Shown on views:** 06, 16, 20

*The provider's file reports an amount the platform cannot tie to anything it believes. Where does that amount go?*

**Context.** Reconciliation never reaches 100%. Files contain lines for transactions the platform has no record of, amounts that differ by a fee the platform calculated differently, and references that match nothing. The convenient treatment is to net the difference into a suspense figure and move on, which is how organisations discover years later that they cannot explain their own cash position. The honest treatment puts the unexplained amount on the books as an explicit liability or asset in a named account, which means the trial balance still balances, the gap is visible, and somebody owns it.

**Decision.** Classify every settlement line as matched, unmatched or disputed. Post unmatched amounts to a named break account rather than discarding or netting them, so the ledger continues to balance and the gap is a row rather than an absence. Age every break, and raise one that passes a threshold — 48 hours by default — as an operational incident rather than a backlog item. Resolve a break only by a new opposite-signed correcting transaction under dual control. Produce a daily trial balance and a cash-position statement, and alarm when the ledger's view of held funds differs from the bank's by more than a configured tolerance.

**How it is realised on this stack.** `settlement_line.txn_id` is nullable and `match_state` is an enum, so an unmatched line is a row carrying its own classification. The matcher posts the unmatched amount against a break account in the same ledger transaction that records the line, preserving the balance invariant. A nightly job ages open breaks and raises a paging alert past 48 hours; break count and break value are both dashboard metrics, with value alarming and count not. Correction flows through the dual-control service in ADR-14.

| Option | Verdict | Reasoning |
|---|---|---|
| Named break account, aged, owned, alarmed; corrections under dual control | Chosen | The trial balance stays meaningful and every unexplained amount has an owner and a clock |
| Net unmatched differences into a suspense figure at period close | Rejected | The default in many organisations and the reason their cash position is not explainable; it converts a findable problem into an unfindable one |
| Reject the file until it matches fully | Rejected | Blocks every payout on the first unexplained line, which at 10 million lines a file means blocking every day |
| Track breaks in a separate operational system outside the ledger | Right elsewhere | Workable where the ledger is a downstream accounting system rather than the system of record; here it would mean the trial balance no longer balances, which removes the one daily check that catches posting bugs |

**What it buys**

- The trial balance balances every day, which turns it into a real check on posting correctness rather than a report
- Every unexplained amount has a name, an age and an owner, so a growing problem is visible within hours instead of at quarter end
- The break account is a leading indicator for provider integration problems that no other metric surfaces
- Resolution is append-only, so the history of what was believed and when is preserved through the correction

**What it costs**

- A standing operational workload — assumed around 40 breaks a day — that must be staffed, and if it is not, the ageing alarm simply becomes noise
- The 48-hour threshold is arbitrary and will be wrong for some providers whose files are routinely restated
- The break account is a real balance-sheet item requiring finance sign-off on its treatment, which is organisational work the architecture creates
- Dual-control correction makes clearing a break slower than editing a row, by design, and that friction will be complained about

**Choose differently when.** If the platform were pure pass-through with no held funds (ADR-13), unmatched lines would be the provider's and the merchant's problem rather than a balance-sheet item, and a simpler exception report might suffice. As custodian of merchant money, the platform owns the gap, and that ownership is what makes the break account necessary rather than merely tidy.

**Why it holds up over time.** "Name the gap rather than netting it" is an accounting discipline, not a technical one. It will apply identically to rails, file formats and reconciliation tooling that do not yet exist, and it is the control that makes every other optimism in this design safe.

> **Lesson.** When two systems disagree about money, the disagreement itself is data worth storing on the books. Netting it away destroys the only record that could ever explain it, and the cost of that destruction is paid years later by someone who cannot reconstruct what happened.

### Shape, scale and contracts

*How state is divided, what is promised to merchants, and how a system holding money is changed.*

#### ADR-08 · Payment state is sharded by merchant; the ledger gives up a global sequence

**Status:** Accepted  ·  **Shown on views:** 08, 11, 18

*What is the partition key, and what ordering guarantee is surrendered to get it?*

**Context.** Two stores need partitioning for different reasons. Payment state is write-heavy and latency-sensitive, and its natural isolation boundary is the merchant: sharding there means one merchant's sale peak cannot consume another's capacity, and every payment-path operation — intent, attempts, refund-sum enforcement — stays inside one shard. The ledger is different. A single global append-only log gives a total order and a trivially correct close of books, and caps throughput at one writer. Partitioning it scales linearly and makes any transaction touching two merchants' accounts a distributed transaction, which is exactly what marketplace splits and inter-merchant transfers require.

**Decision.** Shard payment state by merchant, and keep every payment-path operation inside a single merchant's shard so that no synchronous cross-shard transaction exists. Enforce per-merchant concurrency limits and fair queuing at the edge, so a 10× spike confined to one large merchant does not degrade others. Partition the ledger by fiscal period and account class rather than by merchant, surrendering a single global sequence and providing instead a per-account ordering guarantee and a deterministic close-of-books procedure that aggregates across partitions. Accept that marketplace splits and inter-merchant transfer are therefore out of MVP scope, and state the condition for re-opening this record.

**How it is realised on this stack.** Payment state lives in PostgreSQL shards keyed by a hash of `merchant_id`, with the shard map in the config registry. The edge enforces per-merchant concurrency and a fair queue, so capacity is allocated rather than contested. The ledger is partitioned by period and account class, which keeps a day's postings and a day's close inside a bounded set of partitions and lets closed periods be archived whole. Close of books aggregates per-partition sums and asserts the total is zero; because each transaction's entries are written atomically within one period, the aggregate check is exact rather than approximate.

| Option | Verdict | Reasoning |
|---|---|---|
| Merchant shards for state; period and account-class partitions for the ledger | Chosen | No synchronous cross-shard transaction in the payment path; linear scaling; a deterministic daily close |
| Single global append-only ledger log | Right elsewhere | Right below roughly a thousand transactions a second, where one writer is ample and a total order makes marketplace splits and transfers trivial; wrong at 35 billion entries a year |
| Ledger partitioned by merchant, matching the state store | Rejected | Tempting for symmetry and it makes platform fee revenue — which touches a platform account on every payment — a cross-partition write on every single transaction |
| Ledger partitioned by currency | Deferred | The natural choice once multi-currency settlement lands in Phase 3, and premature in a single-settlement-currency design |

**What it buys**

- One merchant's sale peak is structurally contained: fairness is a property of the partitioning and the queue, not of a throttle somebody has to tune during an incident
- Refund-sum enforcement under concurrency is a single-shard constraint rather than a distributed lock
- Closed fiscal periods archive as whole partitions, which is what keeps a 12 TB-a-year ledger queryable
- Ledger write throughput scales with partitions rather than being capped by one writer

**What it costs**

- There is no global sequence number, so "the Nth transaction ever" is not a concept the platform has
- Close of books is an aggregation across partitions rather than a tail read, and its correctness depends on each transaction being wholly inside one period
- Marketplace splits and inter-merchant transfer become distributed transactions, which is why they are deferred rather than simply unbuilt
- A shard map is operational state: resharding a large merchant is a planned migration, and the design does not make it cheap

**Choose differently when.** This record must be re-opened the moment marketplace splits or inter-merchant transfer are accepted into scope, because both require a transaction spanning two merchants' accounts. If they land, the choice narrows to a two-phase commit across ledger partitions or a repartitioning — and doing it after 35 billion entries exist is far more expensive than doing it now, which is the argument for deciding scope deliberately rather than drifting into it.

**Why it holds up over time.** The merchant as the isolation boundary for payment state is a statement about who shares fate, and that does not change with technology. The ledger's partitioning is more contingent: it is the part of this architecture most likely to be revisited, which is why the conditions for revisiting it are written down.

> **Lesson.** Choose the partition key from the isolation you need rather than from the symmetry you want. Two stores in the same system can and often should be partitioned differently, and matching them for tidiness is how a cross-partition write ends up on every transaction.

#### ADR-09 · Merchant webhooks are at-least-once with a per-object sequence number and a queue per merchant

**Status:** Accepted  ·  **Shown on views:** 05, 17, 09

*What delivery semantics are promised to merchants, and may one slow endpoint delay another merchant's events?*

**Context.** The median merchant integration is written once, by someone with no payments background, and never revisited. It will receive a duplicate webhook and, if it is not idempotent, it will ship the goods twice. Three postures are available. At-least-once with documented merchant-side idempotency is cheap and honest. Ordered-per-object delivery — hold event n+1 until n is acknowledged — is far kinder to naive code and introduces head-of-line blocking that one stalled endpoint turns into a backlog. A pull-only event API moves the problem to the merchant entirely and guarantees that the ones who most need pushing will never poll.

**Decision.** Promise at-least-once delivery explicitly in the contract, and do not claim exactly-once the platform cannot provide. Attach a monotonic per-object sequence number to every event so a merchant can detect a gap and order events for an object without a global order existing. Give every merchant their own delivery queue, so a stalled endpoint delays nobody else. Retry with exponential backoff for up to 72 hours, then expose both an idempotent replay and a pull-based event list with a cursor, so a merchant who was down for a week catches up without contacting support. Auto-disable a persistently failing endpoint with a notice.

**How it is realised on this stack.** The dispatcher consumes the event log and writes into a per-merchant queue, so backpressure is per tenant. Payloads are HMAC-signed with a per-merchant secret that can be rotated with an overlap window, and carry `object_id` and a monotonic `seq` for that object. Retry schedule is exponential to 72 hours. The replay API re-sends by event identifier; the list API pages from a cursor that is the authoritative catch-up mechanism. Delivery attempts are retained for seven years, so "we never got told" is answerable.

| Option | Verdict | Reasoning |
|---|---|---|
| At-least-once, per-object sequence, per-merchant queue | Chosen | Honest contract, gap detection without a global order, and one tenant's failure is contained |
| Ordered-per-object delivery with blocking | Right elsewhere | Right where merchants are few, large and integration quality is contractually managed — it genuinely removes a class of merchant bug; wrong across 50,000 merchants, where it converts one dead endpoint into a shared backlog |
| Pull-only event API, no webhooks | Rejected | Perfectly reliable and guarantees the least sophisticated merchants — the ones who most need the push — never poll |
| Claim exactly-once delivery | Rejected | Not deliverable over HTTP to endpoints that time out after processing, and claiming it means merchants build no defences |

**What it buys**

- A naive but correct merchant handler is possible: deduplicate on event identifier, order on sequence, done — and that is documentable in a paragraph
- One dead endpoint is contained to its own queue, so the common operational incident has a blast radius of one merchant
- A merchant down for a week catches up by cursor without a support ticket or a manual replay
- Stored delivery attempts make the recurring dispute about whether a notification was sent a matter of record

**What it costs**

- Every merchant must be idempotent, and some will not be; duplicate-shipping incidents will happen and will be partly the platform's reputational problem
- Per-merchant queues at 50,000 merchants is a lot of queues to monitor, and queue sprawl is its own operational burden
- No global ordering means a merchant correlating events across objects must do it themselves
- Auto-disable will occasionally silence a merchant who was about to recover, making the notice path as important as the disable

**Choose differently when.** If the merchant base were tens of large, contractually managed integrations rather than 50,000 self-serve ones, ordered delivery would be the better trade: the head-of-line risk shrinks with tenant count and the value of protecting naive handlers rises. The decision rests entirely on the shape of the merchant base.

**Why it holds up over time.** At-least-once plus a per-object sequence is the honest contract for any push over an unreliable transport to an endpoint you do not control. The transport will change; the contract will not, because the underlying impossibility does not.

> **Lesson.** Promise the weakest guarantee you can actually keep, state it prominently, and then give consumers the primitive they need to build the stronger guarantee themselves. A sequence number costs four bytes and removes most of the reason anyone wanted exactly-once.

#### ADR-10 · Posting rules are versioned artefacts, shipped shadowed, and gated on a replay that must be byte-identical

**Status:** Accepted  ·  **Shown on views:** 21, 12, 19

*How is the code that books money changed, given that a posted entry can never be rolled back?*

**Context.** Code rolls back in minutes. Ledger entries do not: a wrong posting is corrected by a new opposite entry, which means a bad release does not merely need reverting, it needs a correction campaign across every transaction it touched, under dual control, with finance sign-off. That asymmetry makes the gate before the release far more valuable than the rollback after it. It also makes the posting rules — the mapping from event type to ledger entries — the highest-risk artefact in the system, and the one most likely to be changed by someone reasoning about accounting rather than about software.

**Decision.** Treat posting rules as versioned, immutable artefacts separate from service code, requiring a finance sign-off to promote. Ship every posting-rule change shadowed first: the new rules are evaluated against live events and their output diffed against the live rules, with nothing booked. Gate promotion on a replay of the last 24 hours of events through the candidate poster producing byte-identical transactions. Run exactly one ledger-poster version at a time, so two rule versions can never book the same event. Use forward-only migrations with no destructive DDL on the append-only tables, and block any change that widens PCI scope at a CDE scope diff.

**How it is realised on this stack.** Rules are content-addressed artefacts in the config registry; `posting_rule_version` is written on every transaction. The shadow poster consumes the same event log with a separate consumer group, writes to a scratch schema, and a differ reports every divergence. The replay gate is a pipeline stage that fails the build on any difference. Canary is 1% of merchants with a 30-minute watch on the correctness metrics from view 19. Image rollback is a normal rolling operation; correction of already-posted entries is a dual-control workflow, deliberately not automated.

| Option | Verdict | Reasoning |
|---|---|---|
| Versioned rules, shadow evaluation, byte-identical replay gate, single poster version | Chosen | The asymmetry between cheap rollback and expensive correction is paid for before the release, not after |
| Posting logic inline in service code, released like any other change | Right elsewhere | Right for a product whose ledger is small enough to correct by hand; wrong where a bad release touches a day's postings across 50,000 merchants |
| Blue/green with two poster versions running | Rejected | Standard practice that is actively wrong here: two versions can book the same event under different rules, which is the exact failure the design exists to prevent |
| Sampled replay instead of a full 24-hour replay | Rejected | The cheap version of the gate, and it misses precisely the rare event types where posting bugs concentrate |

**What it buys**

- A posting-rule bug is caught by the shadow diff before any money is booked, which is the only point at which it is cheap
- The replay gate also proves the natural-key idempotency from ADR-03 continuously, so two guarantees are tested by one mechanism
- `posting_rule_version` on every transaction means a historical entry can be explained by the rules that produced it
- Finance sign-off puts the people accountable for the accounting treatment in the approval path, where they belong

**What it costs**

- The replay gate takes as long as the replay does, and grows with volume — it will become the slowest stage in the pipeline
- The pressure to sample the replay as it slows is the most likely way this control decays, and nothing in the architecture prevents that
- Shadow evaluation doubles posting compute during a rule change, and requires a scratch schema sized for a day of postings
- One poster version at a time means poster deployment is a brief stop-and-start rather than a rolling update, so posting lag spikes on every release

**Choose differently when.** If ledger corrections were cheap — a small merchant base, small amounts, no regulator — the gate would be over-engineered and inline posting logic with a fast rollback would be the right trade. The gate is justified entirely by the cost of correcting a day of postings across 50,000 merchants under dual control.

**Why it holds up over time.** The principle is that the gate's strength should match the cost of the mistake, not the convenience of the release process. That holds for any system where an action cannot be undone, which is a growing rather than shrinking class as more systems take irreversible external actions.

> **Lesson.** Where rollback does not actually undo the damage, spend the effort before the release rather than after. A replay that must be byte-identical is a crude gate and it catches the class of bug that code review and unit tests reliably miss.

### Trust and key custody

*What is inside PCI scope, who may move money, and how large a leaked credential's blast radius is.*

#### ADR-11 · Keep the cardholder data environment small: browser to CDE directly, tokens everywhere else

**Status:** Accepted  ·  **Shown on views:** 11, 22, 13

*Which components are inside PCI-DSS scope, and how few can that be?*

**Context.** Every component that touches a primary account number falls inside PCI-DSS scope, and scope is the dominant long-run cost of accepting cards: it drives segmentation, change control, logging, scanning, penetration testing and an annual assessment for everything inside it. The naive design has card details arriving at the ordinary API gateway, passing through the intent service and out to the acquirer, which puts the entire application estate in scope. The alternative moves the card data path out of the application entirely: the buyer's browser posts card details to a separately clustered proxy inside the CDE, which tokenises and returns a reference, and the application only ever sees the token.

**Decision.** Store no primary account number, CVV or magnetic-stripe equivalent outside the PCI-scoped cardholder data environment, and reference card instruments by network or vault token everywhere else. Route the card data path from the buyer's browser directly into the CDE, bypassing the application zone. Run the CDE as its own cluster with its own HSM pair, and hold all cryptographic key material for tokenisation and provider authentication in HSMs with no key exportable in plaintext. Keep every copy of payment data — including backups and DR — inside the country of operation. Block any change that widens CDE scope at the release gate.

**How it is realised on this stack.** A card data proxy and token vault run on a separate Kubernetes cluster in a dedicated network segment in each data centre, each with a network HSM pair. The checkout SDK posts card details straight to the proxy using a publishable credential plus the intent's client secret; the proxy tokenises and the application receives only a vault or network token. Provider adapters request detokenisation at the moment of the call, inside the segment. The release pipeline computes a CDE scope diff and fails on any new path that could carry a PAN.

| Option | Verdict | Reasoning |
|---|---|---|
| Browser-to-CDE path, separate cluster, HSM key custody, tokens elsewhere | Chosen | Smallest defensible scope; an application-zone compromise yields tokens, not card numbers |
| Card data through the ordinary application path | Rejected | Puts the whole estate in PCI scope, which is the single most expensive avoidable decision available in a card platform |
| Outsource card capture entirely to a provider-hosted page | Right elsewhere | Right for a merchant, and for a payment service that is content to lose control of routing and of the checkout experience; wrong for a platform whose routing across four acquirers is a core capability |
| Encrypt PANs at rest in the application database instead of tokenising | Rejected | Keeps the data in scope while adding key management; tokenisation removes the data, which is the only real reduction |

**What it buys**

- PCI scope is a handful of components rather than the whole estate, which compounds every year in assessment and change-control cost
- A full compromise of the application zone yields tokens and the ability to attempt payments, not the ability to exfiltrate card numbers
- Key material cannot leave the HSM, so an operator with full database access still cannot decrypt anything
- Data residency is satisfied structurally: there is no cloud region to accidentally replicate into

**What it costs**

- A second cluster, a second network segment and HSM pairs in both data centres — real capital and operational cost
- The browser-to-CDE hop is an extra network round trip on the buyer's path, and a CDE outage fails card payments closed
- Detokenisation at call time makes the HSM and vault a hard dependency of every card authorization
- The scope diff in the pipeline will occasionally block a legitimate change and will be experienced as friction

**Choose differently when.** If the platform dropped cards and settled only on account-to-account rails, the CDE, the vault and the HSM pairs would all disappear and this record would be moot. Account-to-account volume is growing fast enough that this is a plausible future, and it would simplify the architecture more than any other single change.

**Why it holds up over time.** "Keep the regulated zone small and let everything else work on a token" is a structural property of where data lives. It will survive PCI-DSS version changes, HSM vendor changes and the arrival of rails nobody has built yet — and it is the reason the blast radius claims in this set are drawn rather than asserted.

> **Lesson.** The cheapest way to secure sensitive data is to arrange for most of your system never to receive it. Scope reduction beats scope hardening, and the architectural move that achieves it — a direct client-to-vault path that bypasses your own application — usually looks like a detour until you cost the alternative.

#### ADR-15 · Publishable and secret credentials are separate, and a leaked secret reaches only its own merchant's account

**Status:** Accepted  ·  **Shown on views:** 05, 22, 09

*A merchant's API credential has leaked. What can the holder do with it?*

**Context.** Merchant credentials leak. They end up in public repositories, in mobile app bundles, in support tickets and in compromised developer laptops, and at 50,000 merchants it is a certainty rather than a risk. Two design properties decide what that costs. First, whether the credential that necessarily reaches the buyer's browser can do anything dangerous — it must exist, because the browser has to confirm a payment, and if it is the same credential the merchant server holds then every merchant with a client-side integration has published their secret. Second, where money can be directed: if a credential can specify a destination account, a leak is a theft; if the destination is a merchant attribute the credential cannot change, a leak is at worst a nuisance.

**Decision.** Issue separate publishable and secret credentials. A publishable credential can do nothing but confirm an existing intent, and cannot create a payment, read a payment, or issue a refund. Scope every credential to a declared set of operations, make each individually revocable, and separate environments. Make the merchant's settlement account a merchant attribute set by the onboarding platform, never a parameter on any API call, so no credential can redirect funds. Assume every secret credential will eventually leak and ensure the holder can only move money into that merchant's own settlement account.

**How it is realised on this stack.** Credentials are per-merchant, per-environment and per-scope, verified at the gateway before any routing. The publishable credential plus the intent's client secret is the only combination the checkout SDK carries, and the intent's amount and merchant are fixed server-side at creation so the browser cannot alter either. Settlement account details live in the merchant record and are changeable only through the onboarding platform under its own dual control. Credential rotation supports an overlap window so rotation never requires downtime.

| Option | Verdict | Reasoning |
|---|---|---|
| Separate publishable and secret credentials, scoped and revocable; destination not a parameter | Chosen | A leaked secret can take money into the victim's own account, which is a nuisance rather than a theft |
| One API key per merchant | Rejected | Forces the secret into the browser for any client-side integration, which publishes it for a large fraction of merchants |
| mTLS client certificates for every merchant | Right elsewhere | Stronger, and right for a small number of large enterprise integrations; wrong for 50,000 self-serve merchants, most of whom cannot manage a certificate |
| Allow the destination account as a request parameter for flexibility | Rejected | Converts every credential leak into a theft, and the flexibility it buys is a use case the onboarding platform should own |

**What it buys**

- A leaked secret credential cannot redirect funds anywhere, which caps the loss at nuisance payments and refunds within one merchant
- The browser never holds anything that can create a payment or read one, so a client-side integration is not a disclosure
- Revocation is per credential, so a leak is remediated without disrupting the merchant's other integrations
- Scoping makes least privilege the default rather than an option merchants have to discover

**What it costs**

- Two credential types to explain, and merchants will occasionally use the wrong one and receive a confusing authorization error
- A leaked secret can still issue refunds within the merchant, which is real damage even if it is not theft
- Settlement account changes require the onboarding platform, which is correct and slower than merchants want
- Credential rotation with overlap windows is more mechanism than a single static key, and some merchants will never rotate

**Choose differently when.** If the merchant base were a handful of large enterprises, mTLS with certificate rotation would be stronger and entirely practical. The split-credential model is chosen for a self-serve base of 50,000, where the binding constraint is what a merchant developer can be relied on to do correctly.

**Why it holds up over time.** The property that matters is that the destination of money is not a parameter the caller controls. That survives every change of credential technology, and it is the difference between a leak that is embarrassing and a leak that is a loss.

> **Lesson.** Design for the credential leak you know is coming by limiting what the credential can mean, not just how well it is protected. Making the dangerous field an attribute of the account rather than a parameter of the request is usually a one-line schema decision with an enormous blast-radius consequence.

### Money, licensing and correction

*Whether the platform holds funds, who may correct the books, and when money is held rather than refused.*

#### ADR-12 · The settlement matcher may resolve payment state, under declared constraint

**Status:** Accepted  ·  **Shown on views:** 14, 16, 20

*If the settlement file says a payment the platform believes failed actually succeeded, may a batch job change the payment's state?*

**Context.** Disagreement between the platform's belief and the provider's file is routine, and the unknown-outcome path in ADR-04 depends on something eventually resolving it. Two options exist. The reconciler corrects payment state itself, which makes a nightly batch job a writer to the payment path and introduces a race with live callbacks. Or it only raises a break for a human, which keeps the payment path single-writer and guarantees a manual backlog that grows linearly with volume — at 12 million attempts a day and a few unknowns per thousand, that is thousands of manual resolutions a day, which is not staffable.

**Decision.** Give the matcher authority to resolve payment state, under four constraints. It transitions state through the same state machine and legal-transition check as any other source, so it cannot produce a state the machine forbids. It is idempotent on the settlement line's identity, so a re-run of a file changes nothing. Its transitions are recorded with `source = reconciliation`, so a resolved payment is distinguishable from one the provider confirmed. And it may only resolve states that are explicitly pending — `processing` and pending refunds — never re-open a terminal `succeeded` or `failed`, which stays a dual-control correction.

**How it is realised on this stack.** The matcher calls the same intent-service transition API that callbacks use, with a source discriminator, rather than writing the payment store directly. Legality is enforced in the state machine, so the matcher's authority is bounded by data rather than by convention. Resolution emits a normal payment event, so the ledger posts through the usual path with the event identifier as natural key, and the merchant webhook fires as it would for any other outcome. A line that implies re-opening a terminal state becomes a break instead (ADR-07).

| Option | Verdict | Reasoning |
|---|---|---|
| Bounded resolution authority through the same state machine | Chosen | Closes the unknown-outcome window automatically while keeping the dangerous transitions human |
| Break-only: the reconciler raises, a human resolves | Right elsewhere | Right at volumes where manual resolution is staffable, and genuinely safer; wrong at thousands a day, where the backlog becomes the failure mode |
| Unbounded resolution authority | Rejected | A batch job able to flip a terminal state is a batch job able to un-refund a buyer, and the blast radius of a matcher bug becomes unbounded |
| Direct writes to the payment store from the matcher | Rejected | Faster to build and bypasses the legality check, which is the only thing making this decision safe |

**What it buys**

- The unknown-outcome window closes without a human, which is what makes ADR-04's 'never retry' rule operationally viable
- Resolution reuses the ordinary transition, event and posting path, so it is exercised by the same tests and metrics as everything else
- `source = reconciliation` makes resolved payments queryable, so the rate of platform-provider disagreement per provider is a measurable number
- The dangerous transitions remain dual-control, so the authority granted is bounded by the state machine rather than by trust

**What it costs**

- There is a real race between the matcher and a late callback arriving for the same payment; it resolves correctly because both go through the legality check, but reasoning about it is genuinely harder
- A matcher bug can now affect payment state, not just reporting — the blast radius is bounded but not zero
- The boundary between 'pending, resolvable' and 'terminal, dual-control' is a judgement call that will need revisiting per provider
- Merchants receive a webhook up to 24 hours after a payment they were told was processing, which some integrations will handle badly

**Choose differently when.** If providers offered reliable inquiry APIs (see ADR-04), the matcher's resolution authority would be needed for a small residue rather than for the main path, and the safer break-only posture would become affordable. The authority exists because the automatic alternative does not.

**Why it holds up over time.** The durable part is the shape of the grant: authority bounded by a state machine rather than by convention, with the source recorded. That pattern — let the batch job act, but only through the same door as everything else — transfers to any system where an asynchronous process must correct a synchronous one.

> **Lesson.** When a background process must write to a path that another process owns, give it the same door rather than a side door. The legality check that already protects the path is what makes the grant safe, and routing the batch job through it costs an API call.

#### ADR-13 · The platform holds merchant balances rather than passing funds through

**Status:** Accepted  ·  **Shown on views:** 15, 16, 12

*Does money rest in the platform's accounts between collection and payout, or does it never stop?*

**Context.** This is the decision that determines perhaps a third of the ledger's account model and most of the platform's regulatory posture, and it is often made implicitly. Pass-through keeps the platform thin: funds settle to the merchant's own account and refund funding and dispute liability sit with the merchant and the acquirer. Holding a balance makes refunds, partial refunds, dispute debits, fee netting, hold periods and marketplace splits all natural operations on an internal account — and makes the platform custodian of other people's money, with segregated client accounts, float accounting, a licence, and a far larger ledger.

**Decision.** Hold merchant balances. Compute each merchant's available balance as settled receipts minus fees, taxes, refunds, disputes and prior payouts, with a configurable hold period per risk tier, and initiate payouts on a per-merchant schedule. Accept the consequences in full: segregated client funds, a named account model including dispute reserve and receivable accounts, float accounting, and the licensing obligations of a custodian. Permit a refund that takes a balance negative and record the result as a receivable rather than rejecting the refund.

**How it is realised on this stack.** The account tree carries merchant payable, platform fee revenue, tax withheld, dispute reserve, settlement clearing, receivable and break accounts, all in the ledger from day one. Available balance is a projection over the ledger with the hold period applied, never a stored number. Hold periods are merchant attributes set by the risk platform, which this service reads. Client funds sit in a segregated settlement account at the bank, and the daily cash-position comparison asserts that the ledger's view of held funds matches it within tolerance.

| Option | Verdict | Reasoning |
|---|---|---|
| Hold merchant balances as custodian | Chosen | Refunds, disputes, fees and hold periods become ledger operations; the cost is licensing and a much larger account model |
| Pure pass-through to the merchant's own account | Right elsewhere | Right for a thin gateway, and much simpler: no client funds, no float, a small ledger; wrong where refunds must succeed regardless of merchant liquidity and disputes must be debitable |
| Hybrid: pass-through with a small retained reserve per merchant | Rejected | Carries most of the custodial obligations for a fraction of the capability, which is the worst of both |
| Defer the decision and build pass-through first | Rejected | The account model is the hardest thing to change after billions of entries exist; deferring it is choosing pass-through without saying so |

**What it buys**

- A refund succeeds even when the merchant has no liquidity, so a buyer who has been charged is never caught by their merchant's cash position
- Dispute debits, fee netting and hold periods are ordinary ledger transactions rather than external arrangements
- Payout timing becomes a platform control, which is what makes risk-tiered hold periods possible at all
- The account model needed for marketplace splits in Phase 3 already exists in outline rather than needing to be retrofitted

**What it costs**

- The platform is a custodian of client money: segregated accounts, float accounting, licensing and a materially heavier compliance obligation
- Negative balances are real credit exposure, and the receivable account will have genuinely uncollectable amounts in it
- Dispute reserve sizing is a risk policy this architecture requires and does not set, which is a dependency on a function outside it
- The ledger is substantially larger than a pass-through design's, which drives the partitioning and archival work in ADR-08

**Choose differently when.** If the licence were unobtainable in the country of operation, or if the business decided the compliance burden outweighed the product value, pass-through would be the only option and roughly a third of the ledger's accounts would disappear along with the payout engine. This is the one record whose reversal would change the shape of every other view in the set.

**Why it holds up over time.** Custody is a business and regulatory posture, not a technical one, and it is the most stable kind of decision there is: it changes when a licence or a business model changes, not when technology does. Which is also why getting it wrong is so expensive.

> **Lesson.** Some architectural decisions are really regulatory ones wearing technical clothing. Decide them explicitly and early, because the data model that follows is the single hardest thing in the system to change later — and deferring the decision is itself a decision.

#### ADR-14 · Dual control is a service in the request path, not a policy in a document

**Status:** Accepted  ·  **Shown on views:** 23, 06, 20

*Can one authenticated human change the books?*

**Context.** Every ledger needs a manual correction path: breaks must be cleared, misapplied fees reversed, genuine disagreements settled. That path is also the most dangerous capability in the platform, and the usual control is a written policy saying two people must agree — enforced by culture, reviewed at audit, and routinely bypassed under time pressure by exactly the people the policy is aimed at. The alternative is to make the second approval a technical precondition, which cannot be skipped because the operation does not complete without it.

**Decision.** Enforce dual control in the path. Any privileged operation that moves money outside the normal flow — manual ledger adjustment, payout release override, merchant balance correction — is a proposal by one authenticated human and an approval by a different authenticated human, with both recorded. Sign the resulting instruction in the HSM so the approval is cryptographic evidence rather than a row somebody with database access could write. Write an immutable audit record for every read and write of payment data by a human operator, including the justifying case reference. Mask amounts and identifiers from operator views by default and require explicit, logged elevation.

**How it is realised on this stack.** A dual-control service holds proposals, enforces that the approver's identity differs from the proposer's, and signs the approved instruction with an HSM key before the ledger accepts it. The ledger's correction path requires a valid signature, so the control is enforced by the store rather than by the console. Workforce authentication requires phishing-resistant MFA. The operator console is masked by default; elevation emits an audit event carrying the support case reference. Audit records are append-only with seven-year retention.

| Option | Verdict | Reasoning |
|---|---|---|
| In-path dual control with HSM-signed instructions | Chosen | No sequence exists in which one human changes the books, and the approval is evidence rather than a claim |
| Policy-based dual control reviewed at audit | Right elsewhere | Right in a small team where the technical control costs more than the risk; wrong anywhere the control is load-bearing, because the bypass is always available exactly when it is most tempting |
| Four-eyes in the ticketing system, single actor in the platform | Rejected | The common compromise: the audit trail looks right and the technical capability is unchanged, so the control is theatre |
| No manual correction path at all | Rejected | Admirably strict and unworkable: breaks and genuine errors need resolution, and removing the path means someone eventually does it with raw SQL |

**What it buys**

- A single compromised operator account cannot move money, which changes the value of phishing a finance team member
- The approval is cryptographic, so it survives a compromise of the application database that could forge an approval row
- Every privileged read is attributable to a case, which makes the insider-curiosity question answerable rather than merely discouraged
- The control is enforced by the ledger, so a new console or a direct API call is subject to it too

**What it costs**

- Clearing a break needs two people available, which is slow at 03:00 and will be complained about in exactly the incidents where speed matters
- A dual-control service on the correction path is another component to keep available, and its unavailability blocks all corrections
- Masking by default makes routine support slower and generates elevation volume that someone must review
- The control is only as strong as the separation between the approvers, which the architecture cannot enforce

**Choose differently when.** If the platform held no funds (ADR-13) and corrections only affected reporting rather than money owed, the HSM signing would be disproportionate and a policy control would be adequate. The cryptographic enforcement is justified by custody: the balances being corrected are other people's money.

**Why it holds up over time.** "Make the control a precondition rather than a rule" outlives every identity provider, HSM and console in the design. The specific mechanism will be replaced; the property that the operation does not complete without a second human will not.

> **Lesson.** A control that can be bypassed under pressure will be bypassed under pressure, and pressure is precisely when it matters. Moving a policy into the request path converts good intentions into a guarantee, and it is usually a smaller build than the policy document was.

#### ADR-17 · A sanctions match holds a payout rather than rejecting it, and payouts move only proven cash

**Status:** Accepted  ·  **Shown on views:** 16, 06, 24

*When screening flags a payout, and when the books have not been proved, what happens to the money?*

**Context.** Two gates sit in front of money leaving the platform and they fail in opposite directions. Sanctions screening produces false positives at a meaningful rate against common names; rejecting on a match would permanently strand a legitimate merchant's funds and create a dispute with no obvious resolution path, while ignoring a match is a regulatory breach. Separately, a payout computed from a ledger that has not been reconciled and proved is a payout of money the platform believes it holds, and the gap between believed and actual cash is exactly what reconciliation exists to find.

**Decision.** Screen payees against sanctions lists before payout and hold rather than reject on a potential match, routing the hold to a review queue with an owner and an ageing clock. Gate the payout run on the daily trial balance and the bank cash-position comparison, so payouts move proven cash rather than believed cash. Treat a payout as pending until the bank confirms, and handle a returned or failed payout by restoring the balance and recording the failure reason. Make payout instructions idempotent on a unique instruction key so a retry cannot send money twice.

**How it is realised on this stack.** Screening runs as a pre-payout step; a match sets the payout to `held` with a reason and raises a review item rather than failing it. The 02:00 close produces the trial balance and the cash-position comparison; a tolerance breach blocks the 06:00 payout run entirely rather than paying out partially, because a partial run on unproven books is harder to unwind than a delay. Payouts carry `UNIQUE (instruction_key)`. A bank return credits the balance back through a normal ledger transaction.

| Option | Verdict | Reasoning |
|---|---|---|
| Hold on match; gate the run on a proved trial balance and bank comparison | Chosen | False positives are recoverable, and no payout moves money the books have not proved |
| Reject on a sanctions match | Rejected | Strands legitimate funds permanently and gives the merchant no path back, which is both a worse outcome and a worse compliance posture than a reviewed hold |
| Pay out on believed balances and reconcile afterwards | Right elsewhere | Right where payout amounts are small relative to capital and speed is the product; wrong as a custodian, where paying out money you do not hold is the failure the licence exists to prevent |
| Block only the matched merchant's payout and run the rest on unproved books | Rejected | Conflates the two gates: the screening gate is per merchant, the proof gate is about the whole ledger |

**What it buys**

- A false positive costs a merchant a delay and a review, not their funds, which is the difference between a support conversation and a legal one
- No payout can exceed what the books have been proved to hold, so the licence obligation is enforced by the pipeline rather than by care
- An idempotent instruction key makes payout retries safe, which is what allows retrying at all
- A returned payout flows back through the ledger, so the merchant's balance history explains the round trip

**What it costs**

- A tolerance breach in the cash comparison blocks every payout that morning, so one unexplained discrepancy delays 50,000 merchants
- Held payouts need a staffed review queue, and an unstaffed queue turns a hold into an indefinite freeze — the outcome the decision was meant to avoid
- Merchants see a balance go up and then back down when a payout is returned, which is honest and confusing
- The tolerance threshold is a judgement call that trades false blocks against undetected drift

**Choose differently when.** If the close-of-books proof could be produced continuously rather than nightly, the all-or-nothing payout gate could become per-merchant and a single discrepancy would stop paying one merchant rather than all of them. That is a worthwhile future change and it depends on the ledger partitioning in ADR-08 supporting a continuous close.

**Why it holds up over time.** Both rules are about the direction a gate fails in: hold rather than reject where a false positive harms an innocent party, and refuse rather than proceed where the books are unproved. Those directions do not change with screening technology or accounting tooling.

> **Lesson.** For every gate in front of an irreversible action, decide which way it fails and why, and check that the failure direction protects the party who cannot protect themselves. A gate whose failure mode nobody chose will fail in whichever direction was easier to code.

### Platform and operations

*Where it runs, what is derived rather than stored, and the cost of the localisation choice.*

#### ADR-16 · Every derived store is rebuildable, and the rebuild is run on a schedule to prove it

**Status:** Accepted  ·  **Shown on views:** 11, 19, 18

*How does the platform know that a balance on a dashboard is derived from the ledger rather than quietly authoritative?*

**Context.** Projections drift into becoming truth. A balance table is created as a cache, an incremental updater is written for speed, a bug or a missed event leaves it slightly wrong, and because nobody ever rebuilds it the drift is permanent and invisible. Eventually a correction is applied to the projection rather than to the ledger, and from that moment the projection holds information the ledger does not. Claiming a store is derived is cheap; proving it requires actually deleting and rebuilding it, and comparing.

**Decision.** Treat every derived store — merchant balances, statements, analytics marts, dashboard caches — as rebuildable from the ledger and the event log, and prove it by rebuilding at least one projection on a schedule and comparing the result. No correction is ever applied to a projection; corrections go to the ledger and flow through. Keep settlement, reporting and analytics workloads off the stores in the payment path, and report their infrastructure cost separately so a reporting workload cannot quietly consume the payment path's budget.

**How it is realised on this stack.** Projection builders are batch jobs running in DC-2 reading the ledger and the event log, writing to versioned target schemas so a rebuild is a build-and-swap rather than a mutation. A scheduled rebuild of the balance projection runs weekly and diffs against the incrementally maintained copy; a non-zero diff is a paging alert, because it means either the incremental path or the rebuild is wrong. Projection stores have no write path from the operator console. Cost attribution tags separate payment-path from reporting infrastructure.

| Option | Verdict | Reasoning |
|---|---|---|
| Rebuildable projections with a scheduled rebuild-and-compare | Chosen | The claim that a store is derived is continuously tested rather than asserted |
| Incrementally maintained projections, trusted | Right elsewhere | Right where projections are cosmetic and a stale number costs nothing; wrong where the number is a merchant's balance and someone will eventually correct it in place |
| Rebuild only on demand, when a discrepancy is reported | Rejected | The rebuild path then has no test coverage and will be broken exactly when it is first needed |
| Compute balances on read from the ledger, no projection | Rejected | Correct by construction and too slow at 35 billion entries; it also puts dashboard load directly on the payment path's store |

**What it buys**

- A drift between the incremental path and the ledger is caught within a week rather than at an audit
- The rebuild path is exercised routinely, so a disaster-recovery rebuild of projections is a known operation rather than an experiment
- No operator write path to projections means the tempting shortcut — fix the number the merchant can see — does not exist
- Reporting load is physically separated from the payment path, so an expensive query cannot slow a payment

**What it costs**

- A weekly full rebuild of a large projection is real compute, and its cost grows with ledger volume
- The rebuild-and-compare job will become slow and will be the first thing someone proposes to disable, which is how this control decays
- Build-and-swap schemas mean projection storage is roughly doubled during a rebuild
- A diff alert is ambiguous — either side could be wrong — so it requires investigation rather than a mechanical fix

**Choose differently when.** If projections were small enough to compute on read, none of this machinery would be needed and the ledger would serve the dashboard directly. At 35 billion entries that is not available, and the rebuild-and-compare is the cheapest substitute for the guarantee that on-read computation would have given for free.

**Why it holds up over time.** "Prove the derivation by performing it" is a general discipline that applies to caches, search indexes, materialised views and anything else claimed to be downstream of a source of truth. The claim decays silently; the scheduled proof does not.

> **Lesson.** A system that claims a store is derived should delete and rebuild it regularly, because the claim is otherwise untested and will eventually be false. The scheduled rebuild is also the only honest way to know the recovery path works.

#### ADR-18 · Self-managed open source in two in-country data centres, not a hyperscaler

**Status:** Accepted  ·  **Shown on views:** 18, 11, 22

*Where does a regulated money path run, and what is the operational price of that answer?*

**Context.** The default answer for a platform of this shape is a managed cloud: fewer components to operate, managed brokers and databases, and an HSM service. Three constraints push against it here. Payment data for domestic transactions must stay in the country, including backups and disaster-recovery copies, which narrows the region choice and makes every managed service's replication topology a compliance question. PCI-DSS scope containment is easier to demonstrate with a physically segmented cardholder data environment than with a shared-tenancy service. And the money path should not depend on a provider the regulator has not assessed. Against that: self-managing Kubernetes, Kafka, PostgreSQL and HSMs is a standing headcount commitment, and an understaffed platform team is the most likely cause of an outage in any design.

**Decision.** Run the platform on self-managed open source across two in-country data centres, 40 km apart at 1.4 ms round-trip. Both sites serve authorization traffic, with DC-2 carrying roughly 30% in steady state so failover is a weight change rather than a cold start. Replicate the ledger synchronously between them for RPO 0. Place settlement, payout and projection batch workloads in DC-2 by default. Accept that there is no third site, and that a metro-wide event is a declared, announced outage with RPO 0 rather than a failover.

**How it is realised on this stack.** Kubernetes with three control-plane nodes per site; Kafka with five brokers, replication factor 3 and min.insync.replicas 2, stretched across both sites; PostgreSQL primary in DC-1 with a synchronous standby in DC-2; network HSM pairs in each site's CDE segment; Envoy at the edge behind an anycast VIP. Provider egress leaves through an allowlisted proxy on static IPs over IPsec and leased lines, which is itself easier to arrange from owned infrastructure than from a cloud egress.

| Option | Verdict | Reasoning |
|---|---|---|
| Self-managed open source, two in-country data centres | Chosen | Residency and PCI scope are structural rather than configured; 40 km buys synchronous replication and RPO 0 |
| Managed cloud in an in-country region | Right elsewhere | Right where the regulator accepts the provider and in-country regions with compliant replication exist — and genuinely cheaper to operate; wrong where every managed service's replication topology becomes a compliance argument |
| Hybrid: CDE on-premises, everything else in a managed cloud | Rejected | A reasonable-sounding compromise that puts the ledger in the cloud and the keys outside it, which maximises the number of residency and latency boundaries on the money path |
| Three sites for metro-event survival | Rejected | The honest fix for the accepted risk, and the third site breaks synchronous replication and therefore RPO 0 — the trade was taken in favour of durability |

**What it buys**

- Data residency is satisfied by there being no other region to replicate into, rather than by configuration anyone could change
- PCI scope is a set of physically segmented racks, which is the easiest form of scope to demonstrate to an assessor
- Synchronous ledger replication at 1.4 ms makes RPO 0 affordable, which is the single strongest durability property in the design
- Static-IP, leased-line egress to money networks is straightforward from owned infrastructure

**What it costs**

- Self-managing Kubernetes, Kafka, PostgreSQL and HSMs is a standing headcount commitment, and the most likely cause of an outage in this design is an understaffed platform team
- Two sites in one metro means a metro-wide event takes both, with no failover available
- Capacity is bought ahead of demand: the 4,000-a-second sale peak must be physically present year round
- Every managed-service convenience — autoscaling groups, managed backups, a console — has to be built or done without

**Choose differently when.** If the regulator accepted a hyperscaler and a compliant in-country region existed with replication confined to the country, the managed option would win on operational cost alone and most of this record would be deleted. It is a regulatory decision presented as an infrastructure one, and it should be re-opened whenever that regulatory position changes.

**Why it holds up over time.** What should outlast the choice is the reasoning: residency, scope containment and dependency approval drove it, not a preference for owned hardware. If those three change, the conclusion should change with them — and the rest of the architecture, which names no vendor in its requirements, is deliberately portable if it does.

> **Lesson.** Record infrastructure decisions with the constraints that forced them, not the preference that resulted. A future team reading "we run our own because of residency, PCI scope and dependency approval" can re-evaluate it when those change; a team reading "we run our own" will either cargo-cult it or rip it out for the wrong reason.

## Every package used, in one table

Twelve terms this set uses precisely, and what each one is deliberately not. Where a more common word exists, the reason it was not used is given — the distinction is usually the one a decision rests on.

| Package | What it is | What it does here | Considered instead |
|---|---|---|---|
| Payment intent | A server-created object representing one attempt to collect a specific amount for a specific merchant, with an explicit state machine and a client secret usable only for it. | The unit of idempotency at the edge, of state, and of everything a buyer can see. | 'Charge' or 'transaction', which imply the money has already moved and leave no name for the period before it has. |
| Attempt | One call to one provider on behalf of one intent, carrying the routing policy version, the normalised result and the raw provider code. | The unit of provider interaction, of settlement matching, and of routing analysis. | Treating the intent as the provider interaction, which makes a second provider attempt indistinguishable from a second payment. |
| `unknown` | A normalised provider result meaning the platform does not know whether the buyer was charged, distinct from approved, soft decline and hard decline. | The outcome that triggers the resolution ladder and forbids any retry. | 'Timeout' or 'error', which both imply nothing happened — the assumption that causes double charges (ADR-04). |
| `processing` | A terminal-pending intent state that only an inbound provider signal or a reconciliation job may leave. | The honest product state for an unresolved payment, reported to buyer and merchant alike. | 'Pending', which is used loosely for everything from 'not started' to 'awaiting capture' and therefore says nothing. |
| Ledger transaction | Two or more entries whose signed minor-unit amounts sum to zero, keyed uniquely on the payment event that caused it. | The unit of money movement, of posting idempotency and of correction. | 'Journal' or 'posting', which in most systems can be edited; this one cannot. |
| Posting rule | A declared, versioned mapping from a payment event type to the ledger entries it produces. | The reason an entry exists, reconstructable years later from the version stamped on the transaction. | Accounting logic embedded in service code, which makes a 2027 entry explainable only by reading 2027's code. |
| Break | A settlement line that cannot be matched to a ledger transaction, posted to a named account so the trial balance still balances. | The named, aged, owned representation of an amount nobody can yet explain. | 'Suspense' or 'unreconciled difference', which are usually netted rather than owned (ADR-07). |
| Trial balance | The daily assertion that the sum of all ledger entries across all partitions is zero, produced by the close-of-books job. | A gate on the payout run, not a report. | A balance report, which tells you numbers rather than telling you whether the numbers can be trusted. |
| Cardholder data environment | The physically and logically segmented zone that is the only place a primary account number may exist, with its own cluster and HSM pair. | The boundary that keeps the rest of the estate out of PCI-DSS scope. | 'Secure zone', which suggests a degree of protection rather than a defined regulatory scope. |
| Available balance | A projection over the ledger of settled receipts minus fees, taxes, refunds, disputes and prior payouts, with the merchant's hold period applied. | What a payout may move, computed rather than stored. | A balance column, which is a number somebody can correct in place — the drift this design exists to prevent (ADR-16). |
| Dual control | A proposal by one authenticated human and an approval by a different one, with the resulting instruction signed in the HSM before the ledger accepts it. | The precondition for every manual money operation. | 'Four eyes' as a policy, which is enforced by culture and bypassed under pressure (ADR-14). |
| Projection | Any store derived from the ledger and the event log, rebuilt on a schedule and compared, with no write path from an operator. | Everything a merchant or an analyst reads, and nothing that holds independent truth. | 'Cache', which implies staleness is the only risk; the real risk is a projection quietly becoming authoritative. |