Architecture Views

When a payment succeeds, does the book of record move inside the buyer's request, or after it?

Context

A payment service has two jobs with incompatible failure profiles. Answering a buyer needs single-digit-hundreds of milliseconds of platform time, tolerates declining, and is worth failing rather than hanging. Keeping the books needs strong consistency, synchronous replication and an append-only store that may never lose or double an entry, and is worth waiting for. Putting both inside one request means the slowest, strictest store in the system sits on the buyer's critical path, so a ledger partition during a sale becomes a checkout outage at 4,000 attempts a second. Separating them means the platform will, for a window, have told a buyer and a merchant that a payment succeeded while the ledger says nothing about it — and that window never fully closes on its own.

Decision

Split the architecture at that boundary. The authorization plane is stateless, deadline-bound, holds no truth and is permitted to decline, time out and fail closed. The ledger plane is append-only, strongly consistent, synchronously replicated, and permitted to be unavailable for writes for minutes. The only coupling between them is a durable ordered event log; no service in the authorization plane writes the ledger, and no ledger operation sits on a buyer request. Accept the resulting window explicitly, make its width a first-class metric, and build the resolution machinery that guarantees it always closes.

How it is realised on this stack

Intent and routing services run as stateless pods on self-managed Kubernetes across both in-country data centres, scaled on concurrent in-flight provider calls rather than request rate. Every state transition is appended to Kafka, partitioned by payment identifier so a single intent's events are strictly ordered, with seven-year retention so the log is evidence rather than a buffer. A dedicated ledger poster is the only consumer that writes PostgreSQL's append-only ledger tables, which replicate synchronously to the second site at 1.4 ms RTT. Posting lag p99 and the age of the oldest unposted event are dashboard metrics with alarms, not debug counters.

Options weighed

• ChosenSeparate planes coupled by a durable ordered log: Checkout survives a ledger partition; the books stay strict; the cost is a visible, bounded, always-closing window
• Right elsewhereSynchronous ledger write inside the buyer's request: Right for a low-volume service where a ledger write is a few milliseconds and no sale peak exists — and it removes an entire class of reconciliation work; wrong at 4,000 attempts a second, where it converts every store hiccup into lost revenue
• RejectedOne service owning both state machine and ledger, posting asynchronously inside itself: Keeps the window but loses the structural guarantee: with both in one process, the first urgent incident produces a direct ledger write and the boundary is gone
• RejectedNo ledger; derive balances from the payment-state table on read: Cheapest to build and impossible to audit: there is no answer to what the platform believed last Tuesday, and corrections become edits

Consequences

What it buys

• A ledger store partition degrades to a lagging balance rather than a failed checkout, which at a sale peak is the difference between a slow dashboard and lost revenue
• The ledger can be as strict as it likes — synchronous replication, balance checks at write, no caching — because nothing in it is latency-critical
• The event log is evidence as well as transport: any projection, including the ledger itself, can be rebuilt by replay
• Reporting, analytics and settlement workloads read stores that are nowhere near the payment path

What it costs

• A permanent window exists in which a payment has succeeded in the world and is not in the books, and in the `unknown` case it can be 24 hours wide
• Four components — the poller, the callback resolver, the settlement matcher and the break manager — exist only to close that window and are pure overhead when the rails behave
• A merchant can be told 'paid' before their balance moves, which generates support contacts that a synchronous design would not
• Two durability stories to operate rather than one: a broker and a relational store, both self-managed

Choose differently when

If volume were an order of magnitude lower and there were no sale peak, a synchronous ledger write inside the request would be simpler, would remove the window entirely, and would delete four components. The decision rests on three facts: a 4,000-a-second peak confined to minutes, a ledger that must be synchronously replicated for RPO 0, and a double-charge budget that makes optimistic retries unavailable. Remove the peak and this record is the wrong one.

Why it holds up over time

The split is a statement about where truth lives, not about Kafka or PostgreSQL. Both can be replaced — and will be — without changing the sentence "the buyer-facing path may fail; the book of record may not be wrong." Every later view in this set either honours that sentence or contradicts it.

What is the authoritative representation of money in this platform — a balance, or a history?

Context

The tempting design is a payments table with a status column and a merchant balance beside it, updated in the same transaction. It is simple, fast, and gives the wrong answer to the only questions that ultimately matter: what did the platform believe about this payment last March, why does this merchant's balance differ from the sum of their payments by 412 rupees, and who changed it. Double-entry bookkeeping answers all three by construction, at the price of roughly eight rows per payment instead of one, an account model that has to be designed before the first payment, and the loss of the ability to simply fix a number.

Decision

Record every money movement as a transaction of two or more entries whose signed amounts sum to zero, against named accounts. Reject an unbalanced transaction at write time rather than correcting it later. Make the ledger append-only — no update, no delete, and no privileged path that bypasses the balance check. Derive every entry from a declared, versioned posting rule keyed on event type. Store amounts as integer minor units with an explicit currency. Every balance, statement and report is a projection of this ledger, and a number that cannot be derived from it is a number the platform does not publish.

How it is realised on this stack

PostgreSQL with `ledger_transaction` and `ledger_entry` tables, a deferred constraint asserting that the signed `amount_minor` of a transaction's entries sums to zero, and `REVOKE UPDATE, DELETE` on both tables for every role including the application's. The named account tree — merchant payable, platform fee revenue, provider receivable, tax withheld, dispute reserve, settlement clearing, break — is reference data with its own history. Posting rules are versioned artefacts in a registry, and `posting_rule_version` is stored on every transaction. Closed fiscal periods are archived at partition level with query support preserved.

Options weighed

• ChosenAppend-only double entry, balanced at write, as sole record: Auditable by construction; corrections are new entries; every report is derivable
• Right elsewherePayment table with a status column and a cached balance: Right for a pre-revenue product or an internal tool where nobody will ever ask what you believed last quarter; wrong the moment a regulator, an auditor or a disputing merchant does
• RejectedSingle-entry transaction log with balances derived by summing: Derivable but not self-checking: nothing catches a posting that credits without a corresponding debit, which is exactly the bug class double entry exists to catch
• RejectedAn off-the-shelf ledger service or blockchain-backed ledger: A managed ledger would be reasonable in a cloud this design cannot use (ADR-18); a blockchain adds an immutability guarantee the platform already has and a consensus cost it does not need

Consequences

What it buys

• An unbalanced transaction is impossible rather than unlikely: the store refuses it, so the bug surfaces as a rejection in the poster rather than as a wrong number in a report
• Any question about history is answerable from the ledger plus the posting-rule version, including years later
• Corrections are new opposite-signed entries, so there is no mechanism by which history changes and therefore no need to trust that nobody used one
• A daily trial balance is a real check rather than a report, because an imbalance means a bug and not a rounding issue

What it costs

• Roughly eight entries per payment — about 35 billion entries a year and 12 TB before compression — which makes partitioning and archival a day-one concern rather than a later one
• The account model must be designed before the first payment, and getting it wrong is expensive once billions of entries reference it
• Nobody can fix a number. Every correction is a workflow, which is correct and is also slower than an operator would like
• Integer minor units mean every interface and every report must agree about currency exponents, and a single float anywhere in the chain is a defect

Choose differently when

If the platform were pure pass-through and never held a balance (see ADR-13), the account model would shrink dramatically and a simpler transaction log might carry it. And if there were no regulatory retention requirement and no disputes, the self-checking property would be worth less than the storage it costs. Neither condition holds here: the platform is custodian of merchant funds and must reconstruct any payment for ten years.

Why it holds up over time

Double-entry bookkeeping predates every technology in this set by centuries and will outlive all of them. Choosing it is choosing not to invent a money model — the one part of this architecture that is certain not to need revisiting, however many times the store beneath it is replaced.

Which duplicate is being prevented — the buyer's second tap, the provider's repeated callback, or the poster's replayed event?

Context

"The system is idempotent" is the sentence that hides the bug. There are at least four distinct duplication events in a payment flow and they need four different keys. A buyer double-tapping produces two HTTP requests carrying the same merchant-supplied `Idempotency-Key`. A provider retrying a webhook produces two callbacks carrying the same provider event identifier. A consumer restart replays events, producing two attempts to post the same ledger transaction. A payout retry produces two instructions to a bank. An edge-only key protects against the first and none of the others; a ledger-only key protects the books and lets duplicate provider calls happen, which is a double charge.

Decision

Declare the natural key at each boundary explicitly and enforce it there. At the edge: the merchant's `Idempotency-Key`, scoped per merchant, where a repeat with an identical body returns the original intent and a repeat with a differing body is a conflict. At the callback: the provider's own event identifier. At the ledger: the payment event's identifier as a unique constraint on the ledger transaction, which is what makes posting exactly-once without coordination. At the payout: a unique instruction key derived from merchant, period and amount. Four keys, four guarantees, documented separately.

How it is realised on this stack

Redis holds edge idempotency keys with a 24-hour TTL, storing a hash of the request body alongside the resulting payment identifier so a conflicting repeat can be detected rather than silently served. Callback dedup is a unique index on `(provider_id, provider_event_id)` in the payment shard. The ledger carries `UNIQUE (event_id)` on `ledger_transaction`, so a replayed event's insert fails on the constraint and the poster treats that failure as success. The payout table carries `UNIQUE (instruction_key)`.

Options weighed

• ChosenA declared natural key at each of the four boundaries: Each duplication event has an owner; replay safety at the ledger needs no distributed coordination
• RejectedOne idempotency key threaded through the whole flow: The merchant's key does not exist at the callback boundary and is not unique per provider event; threading it requires inventing a mapping that is itself a store with its own races
• RejectedExactly-once processing via a distributed transaction across broker and ledger: Buys what a unique constraint already buys, at the cost of a two-phase commit on the ledger's write path
• Right elsewhereDeduplication windows with time-bounded caches at each hop: Right where no stable natural key exists — some legacy rails genuinely have none; wrong here, because a window that expires is a correctness bug waiting for a slow retry

Consequences

What it buys

• The ledger poster can be replayed arbitrarily: a replay of the entire log produces a byte-identical ledger, which is the release gate in ADR-10
• A buyer's double tap is answered with the original intent and produces no second provider call, which is where double charges actually come from
• Each guarantee is independently testable, so a property test can target one boundary at a time
• A conflicting idempotency-key reuse is reported as a conflict rather than quietly creating a second payment — a merchant integration bug surfaces as an error

What it costs

• Four mechanisms to document, test and explain, and a reviewer must understand which one covers which failure
• The edge key store is a hot in-memory dependency on the critical path; its loss degrades duplicate protection even though the ledger stays safe
• The 24-hour edge TTL is a judgement call: a retry at 25 hours creates a second intent, and no window is obviously correct
• Treating a unique-constraint violation as success is correct and reads like a bug to anyone encountering it for the first time

Choose differently when

If every provider offered a reliable idempotency key on authorization requests, the edge key and the no-retry-after-unknown rule in ADR-04 could both relax, because a retry would be provably safe. Most rails do not, and the ones that do disagree about semantics. If that changed industry-wide, this record and ADR-04 would both be rewritten.

Why it holds up over time

The keys are properties of the boundaries, not of the technology at them. A different broker, store or cache changes where the constraint is declared and not which key it is on — and a unique constraint on a natural key will remain the cheapest exactly-once mechanism available for as long as databases have unique constraints.

The provider did not answer within the deadline. Is the buyer charged?

Context

An authorization that times out is not a failure. The issuer may have approved it, the funds may be held, and the buyer's statement may already show it — the platform simply does not know. Treating it as a failure and retrying is the single most reliable way to charge someone twice. Treating it as a success is worse. The honest third state is uncomfortable for everyone: the buyer sees a spinner resolve into 'processing', the merchant cannot decide whether to ship, and the platform has an open obligation it must track to closure. There are three ways out — void-and-retry, poll-until-known, and defer to the settlement file — and they differ by hours and by risk.

Decision

Normalise every provider response into one of four results — `approved`, `soft_decline`, `hard_decline`, `unknown` — and treat `unknown` as a distinct outcome with its own resolution path, never as an error. An intent whose attempt returned `unknown` moves to `processing`, a terminal-pending state that only an inbound provider signal or a reconciliation job may leave. No second provider attempt is ever made after an `unknown`, by any path, including a buyer retry or an operator action. The resolution ladder is provider inquiry where supported, then the daily settlement file. Where a choice exists between declining a legitimate payment and risking a double charge, decline.

How it is realised on this stack

The adapter enforces an 8-second hard deadline and maps timeout, connection reset and any ambiguous provider code to `unknown`, keeping the raw code alongside. The intent state machine declares no legal transition from `processing` whose source is a client request, so a retry is structurally impossible rather than merely discouraged. The outcome poller picks up intents in `processing` beyond 60 seconds and issues provider inquiries with backoff where `provider.supports_inquiry` is true. Unresolved intents are picked up by the settlement matcher (ADR-12). The buyer-facing response for `processing` says explicitly not to retry, and the merchant webhook carries `processing` rather than silence.

Options weighed

• ChosenHold in `processing`; inquire, then defer to the settlement file; never retry: Cannot double-charge; resolves in seconds when inquiry is supported and up to 24 hours when it is not
• Right elsewhereVoid the ambiguous transaction, then retry: Right where every provider reliably honours a void for a transaction it has not acknowledged, which some single-acquirer setups genuinely have; wrong across four providers with differing void semantics, because a void that silently no-ops turns the retry into a double charge
• RejectedRetry against a different provider: The worst option available: the second provider cannot see the first's hold, so a success on both is undetectable until settlement
• RejectedTreat the timeout as a decline and let the buyer try again: Simplest and the classic cause of duplicate charges; it moves the retry decision to the buyer, who has no information

Consequences

What it buys

• The double-charge budget of one per million attempts is achievable, because the action that causes double charges is structurally unavailable
• `processing` is an honest product state the merchant can reason about, rather than a lie in either direction
• Buyer retries are free: the edge idempotency key returns the same intent and no new attempt is made
• The resolution path is the same machinery as ordinary reconciliation, so it is exercised daily rather than only during incidents

What it costs

• Resolution can take up to 24 hours where the provider supports no inquiry, which is a genuinely poor buyer experience
• Some merchants will ship on `processing` despite being told not to, and some of those payments will resolve as declined
• A provider outage produces a growing population of `processing` intents, which is operationally alarming even when it is working as designed
• Conversion is lost: a buyer told 'processing' sometimes abandons and pays by another method elsewhere

Choose differently when

If every provider in the routing set supported a reliable inquiry API answering within seconds, the 24-hour tail would disappear and `processing` would become a sub-second state — at which point void-and-retry becomes unnecessary rather than merely risky. Conversely, if the business decided that a double charge refunded within the hour was cheaper than lost conversion, the retry ban would be the wrong rule, and this record would be the first to go.

Why it holds up over time

New rails arrive constantly and every one of them will occasionally fail to say what happened. A design that treats ambiguity as a first-class state absorbs a new rail as an adapter; one that treats it as an error rewrites itself for each. The rule survives every change of provider, protocol and decade.

When the provider's API and the provider's settlement file disagree, which one does the platform believe?

Context

A provider speaks to the platform through at least three channels that routinely disagree: a synchronous API response, asynchronous callbacks, and a daily settlement file. The API response is fastest and least reliable — it can time out after approval, report success for a transaction later reversed, or carry a code whose meaning differs between providers. Callbacks arrive duplicated, out of order, hours late, or occasionally for transactions the platform has no record of. The settlement file is slow, sometimes restated, and is the thing the provider will actually pay against. A platform that treats the API response as truth books revenue it may not receive; one that waits for the file cannot answer a buyer.

Decision

Rank the channels explicitly. The synchronous response advances the payment state machine and may be reported to the buyer and merchant. The callback resolves ambiguity and may advance state, subject to deduplication and a legal-transition check. The settlement file is the evidence: it is what reconciliation matches against, what payouts are computed from, and what wins any disagreement. Persist every raw callback payload before interpreting it, so a parser bug destroys no evidence, and ignore — while recording — any callback whose implied transition is illegal from the current state.

How it is realised on this stack

The callback endpoint verifies signature and freshness (5-minute window, nonce replay rejected) at the perimeter, writes the raw body to a 7-year payload store keyed by a content hash, and only then parses. Deduplication is a unique index on `(provider_id, provider_event_id)`. The state resolver consults the intent's declared legal transitions and records an `ignored_transition` event rather than dropping a callback silently. Settlement files land on SFTP, are hashed and recorded with a run identifier, and matched on `provider_ref` plus amount against `payment_attempt`.

Options weighed

• ChosenRanked channels: response advances, callback resolves, file decides: Fast enough for a buyer, authoritative where money is concerned, and the ranking is explicit rather than emergent
• Right elsewhereTreat the synchronous response as truth and reconcile as a back-office exercise: Right for a single-acquirer, low-volume integration with a reliable partner and small amounts; wrong where settlement disagreement is routine and the platform pays merchants before it is paid
• RejectedWait for the settlement file before reporting any outcome: Correct and unusable: no buyer waits 24 hours, and no merchant can run a storefront on T+1 outcomes
• RejectedTrust whichever channel arrived most recently: Not a ranking at all; with out-of-order callbacks it makes the final state depend on network timing

Consequences

What it buys

• Revenue is only ever recognised against evidence the provider will pay, so the books do not drift optimistically
• A parser bug or a schema change is recoverable: the raw payloads are kept and can be reprocessed
• Out-of-order and duplicate callbacks are one mechanism — dedup plus legality — rather than four separate races
• An ignored callback is recorded, so 'we never got told' is an answerable question

What it costs

• Three channels to integrate, monitor and version per provider, each with its own failure modes
• Raw payload storage for seven years is a meaningful volume and a data-residency obligation in its own right
• Buyer-visible state can be contradicted later by the file, which is honest and generates support work
• A restated file means yesterday's reconciliation output can change, so no reconciliation result is ever final until the period closes

Choose differently when

If the platform settled directly on a rail it operated, there would be no external file to reconcile against and the API response would be the truth by definition. That is the position of a network rather than a payment service, and it is explicitly out of scope. Within scope, the ranking would only change if a provider offered a synchronous response with a contractual settlement guarantee attached — which none does.

Why it holds up over time

The ranking is about trust, not technology. Protocols will change from SFTP files to streaming APIs and the question — which channel will the counterparty actually pay against — will have the same answer and the same ranking.

Why did this payment go to this acquirer, and can that be answered in three years?

Context

With four providers across two rails, every attempt needs a routing decision, and that decision has commercial, reliability and regulatory consequences. The options run from static per-method configuration (auditable and dumb) through a rules engine evaluated per attempt (expressive, and a second production system with its own correctness problem) to an outcome-learning optimiser (highest value, and nearly impossible to explain to an auditor asking why this payment went where it did). Underneath sits a harder requirement: a disputing merchant or a regulator may ask about a specific payment years later, and "the model decided" is not an answer.

Decision

Express routing as a declarative policy evaluating payment method, issuer or bank identifier, amount band, merchant preference, provider health and provider cost. Version the policy as an immutable artefact and store `policy_version` on every attempt, so any decision is reproducible from stored inputs. Make cost an input that is strictly subordinate to approval rate and to any merchant-declared constraint. Derive provider health from observed approval rate and latency, never from a provider's status page. Allow providers to be added, weighted, drained and removed by configuration change, without deploying the authorization path. Defer outcome-learning routing to Phase 3, and only with reproducible decision records.

How it is realised on this stack

Policies are YAML artefacts in a registry, content-addressed and promoted through the same pipeline as code with a required commercial sign-off. Pods hold the active policy in memory and reload on registry change; the active version identifier is stamped on every `payment_attempt` row alongside the raw provider code. Health is computed from a rolling window of observed outcomes per provider and per method, feeding both the circuit breaker and the policy's health input. Draining a provider is a weight change, not a deployment.

Options weighed

• ChosenVersioned declarative policy, decisions reproducible from stored inputs: Expressive enough for real routing, auditable years later, changeable without a deployment
• Right elsewhereStatic per-method configuration: Right with one or two providers and no commercial routing requirement, where the simplicity is worth more than the expressiveness; wrong with four providers and per-issuer approval differences worth real money
• DeferredOutcome-learning cost-and-approval optimiser: The highest-value option and the hardest to explain; revisit in Phase 3 once decision records are reproducible, which is what this record puts in place
• RejectedLet each merchant specify the provider: Hands routing risk to people with no visibility of provider health, and makes draining a failing provider a customer-communication exercise

Consequences

What it buys

• Any routing decision is reconstructable from the stored policy version and the attempt row, which is what makes a regulator's question answerable
• Draining a degraded provider is a configuration change evaluated on the next attempt, not a release
• Health derived from observed behaviour reacts faster than a provider's own status reporting and is not subject to its incentives
• Cost being structurally subordinate to approval rate prevents the common failure where routing quietly optimises margin into lost conversion

What it costs

• A policy language is a second production system with its own validation, testing and review needs
• Immutable policy versions accumulate, and reproducing a 2027 decision in 2030 means keeping the evaluator able to run a 2027 policy
• Declarative routing leaves value on the table compared with a learned router, and that gap grows with provider count
• A policy change with a sign-off gate is slower than a commercial team would like, which creates pressure to add overrides

Choose differently when

If the platform ever ran with a single acquirer, the whole policy mechanism would be dead weight and static configuration would be correct. In the other direction, once the provider set grows past roughly ten and per-issuer approval differences dominate, declarative routing stops paying and the learned router in Phase 3 becomes the right answer — carrying this record's reproducibility requirement with it.

Why it holds up over time

The durable part is not the policy language but the requirement that a decision is reproducible from stored inputs. That requirement survives the move to a learned router, which is exactly why it is written down before the router exists rather than after.

The provider's file reports an amount the platform cannot tie to anything it believes. Where does that amount go?

Context

Reconciliation never reaches 100%. Files contain lines for transactions the platform has no record of, amounts that differ by a fee the platform calculated differently, and references that match nothing. The convenient treatment is to net the difference into a suspense figure and move on, which is how organisations discover years later that they cannot explain their own cash position. The honest treatment puts the unexplained amount on the books as an explicit liability or asset in a named account, which means the trial balance still balances, the gap is visible, and somebody owns it.

Decision

Classify every settlement line as matched, unmatched or disputed. Post unmatched amounts to a named break account rather than discarding or netting them, so the ledger continues to balance and the gap is a row rather than an absence. Age every break, and raise one that passes a threshold — 48 hours by default — as an operational incident rather than a backlog item. Resolve a break only by a new opposite-signed correcting transaction under dual control. Produce a daily trial balance and a cash-position statement, and alarm when the ledger's view of held funds differs from the bank's by more than a configured tolerance.

How it is realised on this stack

`settlement_line.txn_id` is nullable and `match_state` is an enum, so an unmatched line is a row carrying its own classification. The matcher posts the unmatched amount against a break account in the same ledger transaction that records the line, preserving the balance invariant. A nightly job ages open breaks and raises a paging alert past 48 hours; break count and break value are both dashboard metrics, with value alarming and count not. Correction flows through the dual-control service in ADR-14.

Options weighed

• ChosenNamed break account, aged, owned, alarmed; corrections under dual control: The trial balance stays meaningful and every unexplained amount has an owner and a clock
• RejectedNet unmatched differences into a suspense figure at period close: The default in many organisations and the reason their cash position is not explainable; it converts a findable problem into an unfindable one
• RejectedReject the file until it matches fully: Blocks every payout on the first unexplained line, which at 10 million lines a file means blocking every day
• Right elsewhereTrack breaks in a separate operational system outside the ledger: Workable where the ledger is a downstream accounting system rather than the system of record; here it would mean the trial balance no longer balances, which removes the one daily check that catches posting bugs

Consequences

What it buys

• The trial balance balances every day, which turns it into a real check on posting correctness rather than a report
• Every unexplained amount has a name, an age and an owner, so a growing problem is visible within hours instead of at quarter end
• The break account is a leading indicator for provider integration problems that no other metric surfaces
• Resolution is append-only, so the history of what was believed and when is preserved through the correction

What it costs

• A standing operational workload — assumed around 40 breaks a day — that must be staffed, and if it is not, the ageing alarm simply becomes noise
• The 48-hour threshold is arbitrary and will be wrong for some providers whose files are routinely restated
• The break account is a real balance-sheet item requiring finance sign-off on its treatment, which is organisational work the architecture creates
• Dual-control correction makes clearing a break slower than editing a row, by design, and that friction will be complained about

Choose differently when

If the platform were pure pass-through with no held funds (ADR-13), unmatched lines would be the provider's and the merchant's problem rather than a balance-sheet item, and a simpler exception report might suffice. As custodian of merchant money, the platform owns the gap, and that ownership is what makes the break account necessary rather than merely tidy.

Why it holds up over time

"Name the gap rather than netting it" is an accounting discipline, not a technical one. It will apply identically to rails, file formats and reconciliation tooling that do not yet exist, and it is the control that makes every other optimism in this design safe.

What is the partition key, and what ordering guarantee is surrendered to get it?

Context

Two stores need partitioning for different reasons. Payment state is write-heavy and latency-sensitive, and its natural isolation boundary is the merchant: sharding there means one merchant's sale peak cannot consume another's capacity, and every payment-path operation — intent, attempts, refund-sum enforcement — stays inside one shard. The ledger is different. A single global append-only log gives a total order and a trivially correct close of books, and caps throughput at one writer. Partitioning it scales linearly and makes any transaction touching two merchants' accounts a distributed transaction, which is exactly what marketplace splits and inter-merchant transfers require.

Decision

Shard payment state by merchant, and keep every payment-path operation inside a single merchant's shard so that no synchronous cross-shard transaction exists. Enforce per-merchant concurrency limits and fair queuing at the edge, so a 10× spike confined to one large merchant does not degrade others. Partition the ledger by fiscal period and account class rather than by merchant, surrendering a single global sequence and providing instead a per-account ordering guarantee and a deterministic close-of-books procedure that aggregates across partitions. Accept that marketplace splits and inter-merchant transfer are therefore out of MVP scope, and state the condition for re-opening this record.

How it is realised on this stack

Payment state lives in PostgreSQL shards keyed by a hash of `merchant_id`, with the shard map in the config registry. The edge enforces per-merchant concurrency and a fair queue, so capacity is allocated rather than contested. The ledger is partitioned by period and account class, which keeps a day's postings and a day's close inside a bounded set of partitions and lets closed periods be archived whole. Close of books aggregates per-partition sums and asserts the total is zero; because each transaction's entries are written atomically within one period, the aggregate check is exact rather than approximate.

Options weighed

• ChosenMerchant shards for state; period and account-class partitions for the ledger: No synchronous cross-shard transaction in the payment path; linear scaling; a deterministic daily close
• Right elsewhereSingle global append-only ledger log: Right below roughly a thousand transactions a second, where one writer is ample and a total order makes marketplace splits and transfers trivial; wrong at 35 billion entries a year
• RejectedLedger partitioned by merchant, matching the state store: Tempting for symmetry and it makes platform fee revenue — which touches a platform account on every payment — a cross-partition write on every single transaction
• DeferredLedger partitioned by currency: The natural choice once multi-currency settlement lands in Phase 3, and premature in a single-settlement-currency design

Consequences

What it buys

• One merchant's sale peak is structurally contained: fairness is a property of the partitioning and the queue, not of a throttle somebody has to tune during an incident
• Refund-sum enforcement under concurrency is a single-shard constraint rather than a distributed lock
• Closed fiscal periods archive as whole partitions, which is what keeps a 12 TB-a-year ledger queryable
• Ledger write throughput scales with partitions rather than being capped by one writer

What it costs

• There is no global sequence number, so "the Nth transaction ever" is not a concept the platform has
• Close of books is an aggregation across partitions rather than a tail read, and its correctness depends on each transaction being wholly inside one period
• Marketplace splits and inter-merchant transfer become distributed transactions, which is why they are deferred rather than simply unbuilt
• A shard map is operational state: resharding a large merchant is a planned migration, and the design does not make it cheap

Choose differently when

This record must be re-opened the moment marketplace splits or inter-merchant transfer are accepted into scope, because both require a transaction spanning two merchants' accounts. If they land, the choice narrows to a two-phase commit across ledger partitions or a repartitioning — and doing it after 35 billion entries exist is far more expensive than doing it now, which is the argument for deciding scope deliberately rather than drifting into it.

Why it holds up over time

The merchant as the isolation boundary for payment state is a statement about who shares fate, and that does not change with technology. The ledger's partitioning is more contingent: it is the part of this architecture most likely to be revisited, which is why the conditions for revisiting it are written down.

What delivery semantics are promised to merchants, and may one slow endpoint delay another merchant's events?

Context

The median merchant integration is written once, by someone with no payments background, and never revisited. It will receive a duplicate webhook and, if it is not idempotent, it will ship the goods twice. Three postures are available. At-least-once with documented merchant-side idempotency is cheap and honest. Ordered-per-object delivery — hold event n+1 until n is acknowledged — is far kinder to naive code and introduces head-of-line blocking that one stalled endpoint turns into a backlog. A pull-only event API moves the problem to the merchant entirely and guarantees that the ones who most need pushing will never poll.

Decision

Promise at-least-once delivery explicitly in the contract, and do not claim exactly-once the platform cannot provide. Attach a monotonic per-object sequence number to every event so a merchant can detect a gap and order events for an object without a global order existing. Give every merchant their own delivery queue, so a stalled endpoint delays nobody else. Retry with exponential backoff for up to 72 hours, then expose both an idempotent replay and a pull-based event list with a cursor, so a merchant who was down for a week catches up without contacting support. Auto-disable a persistently failing endpoint with a notice.

How it is realised on this stack

The dispatcher consumes the event log and writes into a per-merchant queue, so backpressure is per tenant. Payloads are HMAC-signed with a per-merchant secret that can be rotated with an overlap window, and carry `object_id` and a monotonic `seq` for that object. Retry schedule is exponential to 72 hours. The replay API re-sends by event identifier; the list API pages from a cursor that is the authoritative catch-up mechanism. Delivery attempts are retained for seven years, so "we never got told" is answerable.

Options weighed

• ChosenAt-least-once, per-object sequence, per-merchant queue: Honest contract, gap detection without a global order, and one tenant's failure is contained
• Right elsewhereOrdered-per-object delivery with blocking: Right where merchants are few, large and integration quality is contractually managed — it genuinely removes a class of merchant bug; wrong across 50,000 merchants, where it converts one dead endpoint into a shared backlog
• RejectedPull-only event API, no webhooks: Perfectly reliable and guarantees the least sophisticated merchants — the ones who most need the push — never poll
• RejectedClaim exactly-once delivery: Not deliverable over HTTP to endpoints that time out after processing, and claiming it means merchants build no defences

Consequences

What it buys

• A naive but correct merchant handler is possible: deduplicate on event identifier, order on sequence, done — and that is documentable in a paragraph
• One dead endpoint is contained to its own queue, so the common operational incident has a blast radius of one merchant
• A merchant down for a week catches up by cursor without a support ticket or a manual replay
• Stored delivery attempts make the recurring dispute about whether a notification was sent a matter of record

What it costs

• Every merchant must be idempotent, and some will not be; duplicate-shipping incidents will happen and will be partly the platform's reputational problem
• Per-merchant queues at 50,000 merchants is a lot of queues to monitor, and queue sprawl is its own operational burden
• No global ordering means a merchant correlating events across objects must do it themselves
• Auto-disable will occasionally silence a merchant who was about to recover, making the notice path as important as the disable

Choose differently when

If the merchant base were tens of large, contractually managed integrations rather than 50,000 self-serve ones, ordered delivery would be the better trade: the head-of-line risk shrinks with tenant count and the value of protecting naive handlers rises. The decision rests entirely on the shape of the merchant base.

Why it holds up over time

At-least-once plus a per-object sequence is the honest contract for any push over an unreliable transport to an endpoint you do not control. The transport will change; the contract will not, because the underlying impossibility does not.

How is the code that books money changed, given that a posted entry can never be rolled back?

Context

Code rolls back in minutes. Ledger entries do not: a wrong posting is corrected by a new opposite entry, which means a bad release does not merely need reverting, it needs a correction campaign across every transaction it touched, under dual control, with finance sign-off. That asymmetry makes the gate before the release far more valuable than the rollback after it. It also makes the posting rules — the mapping from event type to ledger entries — the highest-risk artefact in the system, and the one most likely to be changed by someone reasoning about accounting rather than about software.

Decision

Treat posting rules as versioned, immutable artefacts separate from service code, requiring a finance sign-off to promote. Ship every posting-rule change shadowed first: the new rules are evaluated against live events and their output diffed against the live rules, with nothing booked. Gate promotion on a replay of the last 24 hours of events through the candidate poster producing byte-identical transactions. Run exactly one ledger-poster version at a time, so two rule versions can never book the same event. Use forward-only migrations with no destructive DDL on the append-only tables, and block any change that widens PCI scope at a CDE scope diff.

How it is realised on this stack

Rules are content-addressed artefacts in the config registry; `posting_rule_version` is written on every transaction. The shadow poster consumes the same event log with a separate consumer group, writes to a scratch schema, and a differ reports every divergence. The replay gate is a pipeline stage that fails the build on any difference. Canary is 1% of merchants with a 30-minute watch on the correctness metrics from view 19. Image rollback is a normal rolling operation; correction of already-posted entries is a dual-control workflow, deliberately not automated.

Options weighed

• ChosenVersioned rules, shadow evaluation, byte-identical replay gate, single poster version: The asymmetry between cheap rollback and expensive correction is paid for before the release, not after
• Right elsewherePosting logic inline in service code, released like any other change: Right for a product whose ledger is small enough to correct by hand; wrong where a bad release touches a day's postings across 50,000 merchants
• RejectedBlue/green with two poster versions running: Standard practice that is actively wrong here: two versions can book the same event under different rules, which is the exact failure the design exists to prevent
• RejectedSampled replay instead of a full 24-hour replay: The cheap version of the gate, and it misses precisely the rare event types where posting bugs concentrate

Consequences

What it buys

• A posting-rule bug is caught by the shadow diff before any money is booked, which is the only point at which it is cheap
• The replay gate also proves the natural-key idempotency from ADR-03 continuously, so two guarantees are tested by one mechanism
• `posting_rule_version` on every transaction means a historical entry can be explained by the rules that produced it
• Finance sign-off puts the people accountable for the accounting treatment in the approval path, where they belong

What it costs

• The replay gate takes as long as the replay does, and grows with volume — it will become the slowest stage in the pipeline
• The pressure to sample the replay as it slows is the most likely way this control decays, and nothing in the architecture prevents that
• Shadow evaluation doubles posting compute during a rule change, and requires a scratch schema sized for a day of postings
• One poster version at a time means poster deployment is a brief stop-and-start rather than a rolling update, so posting lag spikes on every release

Choose differently when

If ledger corrections were cheap — a small merchant base, small amounts, no regulator — the gate would be over-engineered and inline posting logic with a fast rollback would be the right trade. The gate is justified entirely by the cost of correcting a day of postings across 50,000 merchants under dual control.

Why it holds up over time

The principle is that the gate's strength should match the cost of the mistake, not the convenience of the release process. That holds for any system where an action cannot be undone, which is a growing rather than shrinking class as more systems take irreversible external actions.

Which components are inside PCI-DSS scope, and how few can that be?

Context

Every component that touches a primary account number falls inside PCI-DSS scope, and scope is the dominant long-run cost of accepting cards: it drives segmentation, change control, logging, scanning, penetration testing and an annual assessment for everything inside it. The naive design has card details arriving at the ordinary API gateway, passing through the intent service and out to the acquirer, which puts the entire application estate in scope. The alternative moves the card data path out of the application entirely: the buyer's browser posts card details to a separately clustered proxy inside the CDE, which tokenises and returns a reference, and the application only ever sees the token.

Decision

Store no primary account number, CVV or magnetic-stripe equivalent outside the PCI-scoped cardholder data environment, and reference card instruments by network or vault token everywhere else. Route the card data path from the buyer's browser directly into the CDE, bypassing the application zone. Run the CDE as its own cluster with its own HSM pair, and hold all cryptographic key material for tokenisation and provider authentication in HSMs with no key exportable in plaintext. Keep every copy of payment data — including backups and DR — inside the country of operation. Block any change that widens CDE scope at the release gate.

How it is realised on this stack

A card data proxy and token vault run on a separate Kubernetes cluster in a dedicated network segment in each data centre, each with a network HSM pair. The checkout SDK posts card details straight to the proxy using a publishable credential plus the intent's client secret; the proxy tokenises and the application receives only a vault or network token. Provider adapters request detokenisation at the moment of the call, inside the segment. The release pipeline computes a CDE scope diff and fails on any new path that could carry a PAN.

Options weighed

• ChosenBrowser-to-CDE path, separate cluster, HSM key custody, tokens elsewhere: Smallest defensible scope; an application-zone compromise yields tokens, not card numbers
• RejectedCard data through the ordinary application path: Puts the whole estate in PCI scope, which is the single most expensive avoidable decision available in a card platform
• Right elsewhereOutsource card capture entirely to a provider-hosted page: Right for a merchant, and for a payment service that is content to lose control of routing and of the checkout experience; wrong for a platform whose routing across four acquirers is a core capability
• RejectedEncrypt PANs at rest in the application database instead of tokenising: Keeps the data in scope while adding key management; tokenisation removes the data, which is the only real reduction

Consequences

What it buys

• PCI scope is a handful of components rather than the whole estate, which compounds every year in assessment and change-control cost
• A full compromise of the application zone yields tokens and the ability to attempt payments, not the ability to exfiltrate card numbers
• Key material cannot leave the HSM, so an operator with full database access still cannot decrypt anything
• Data residency is satisfied structurally: there is no cloud region to accidentally replicate into

What it costs

• A second cluster, a second network segment and HSM pairs in both data centres — real capital and operational cost
• The browser-to-CDE hop is an extra network round trip on the buyer's path, and a CDE outage fails card payments closed
• Detokenisation at call time makes the HSM and vault a hard dependency of every card authorization
• The scope diff in the pipeline will occasionally block a legitimate change and will be experienced as friction

Choose differently when

If the platform dropped cards and settled only on account-to-account rails, the CDE, the vault and the HSM pairs would all disappear and this record would be moot. Account-to-account volume is growing fast enough that this is a plausible future, and it would simplify the architecture more than any other single change.

Why it holds up over time

"Keep the regulated zone small and let everything else work on a token" is a structural property of where data lives. It will survive PCI-DSS version changes, HSM vendor changes and the arrival of rails nobody has built yet — and it is the reason the blast radius claims in this set are drawn rather than asserted.

A merchant's API credential has leaked. What can the holder do with it?

Context

Merchant credentials leak. They end up in public repositories, in mobile app bundles, in support tickets and in compromised developer laptops, and at 50,000 merchants it is a certainty rather than a risk. Two design properties decide what that costs. First, whether the credential that necessarily reaches the buyer's browser can do anything dangerous — it must exist, because the browser has to confirm a payment, and if it is the same credential the merchant server holds then every merchant with a client-side integration has published their secret. Second, where money can be directed: if a credential can specify a destination account, a leak is a theft; if the destination is a merchant attribute the credential cannot change, a leak is at worst a nuisance.

Decision

Issue separate publishable and secret credentials. A publishable credential can do nothing but confirm an existing intent, and cannot create a payment, read a payment, or issue a refund. Scope every credential to a declared set of operations, make each individually revocable, and separate environments. Make the merchant's settlement account a merchant attribute set by the onboarding platform, never a parameter on any API call, so no credential can redirect funds. Assume every secret credential will eventually leak and ensure the holder can only move money into that merchant's own settlement account.

How it is realised on this stack

Credentials are per-merchant, per-environment and per-scope, verified at the gateway before any routing. The publishable credential plus the intent's client secret is the only combination the checkout SDK carries, and the intent's amount and merchant are fixed server-side at creation so the browser cannot alter either. Settlement account details live in the merchant record and are changeable only through the onboarding platform under its own dual control. Credential rotation supports an overlap window so rotation never requires downtime.

Options weighed

• ChosenSeparate publishable and secret credentials, scoped and revocable; destination not a parameter: A leaked secret can take money into the victim's own account, which is a nuisance rather than a theft
• RejectedOne API key per merchant: Forces the secret into the browser for any client-side integration, which publishes it for a large fraction of merchants
• Right elsewheremTLS client certificates for every merchant: Stronger, and right for a small number of large enterprise integrations; wrong for 50,000 self-serve merchants, most of whom cannot manage a certificate
• RejectedAllow the destination account as a request parameter for flexibility: Converts every credential leak into a theft, and the flexibility it buys is a use case the onboarding platform should own

Consequences

What it buys

• A leaked secret credential cannot redirect funds anywhere, which caps the loss at nuisance payments and refunds within one merchant
• The browser never holds anything that can create a payment or read one, so a client-side integration is not a disclosure
• Revocation is per credential, so a leak is remediated without disrupting the merchant's other integrations
• Scoping makes least privilege the default rather than an option merchants have to discover

What it costs

• Two credential types to explain, and merchants will occasionally use the wrong one and receive a confusing authorization error
• A leaked secret can still issue refunds within the merchant, which is real damage even if it is not theft
• Settlement account changes require the onboarding platform, which is correct and slower than merchants want
• Credential rotation with overlap windows is more mechanism than a single static key, and some merchants will never rotate

Choose differently when

If the merchant base were a handful of large enterprises, mTLS with certificate rotation would be stronger and entirely practical. The split-credential model is chosen for a self-serve base of 50,000, where the binding constraint is what a merchant developer can be relied on to do correctly.

Why it holds up over time

The property that matters is that the destination of money is not a parameter the caller controls. That survives every change of credential technology, and it is the difference between a leak that is embarrassing and a leak that is a loss.

If the settlement file says a payment the platform believes failed actually succeeded, may a batch job change the payment's state?

Context

Disagreement between the platform's belief and the provider's file is routine, and the unknown-outcome path in ADR-04 depends on something eventually resolving it. Two options exist. The reconciler corrects payment state itself, which makes a nightly batch job a writer to the payment path and introduces a race with live callbacks. Or it only raises a break for a human, which keeps the payment path single-writer and guarantees a manual backlog that grows linearly with volume — at 12 million attempts a day and a few unknowns per thousand, that is thousands of manual resolutions a day, which is not staffable.

Decision

Give the matcher authority to resolve payment state, under four constraints. It transitions state through the same state machine and legal-transition check as any other source, so it cannot produce a state the machine forbids. It is idempotent on the settlement line's identity, so a re-run of a file changes nothing. Its transitions are recorded with `source = reconciliation`, so a resolved payment is distinguishable from one the provider confirmed. And it may only resolve states that are explicitly pending — `processing` and pending refunds — never re-open a terminal `succeeded` or `failed`, which stays a dual-control correction.

How it is realised on this stack

The matcher calls the same intent-service transition API that callbacks use, with a source discriminator, rather than writing the payment store directly. Legality is enforced in the state machine, so the matcher's authority is bounded by data rather than by convention. Resolution emits a normal payment event, so the ledger posts through the usual path with the event identifier as natural key, and the merchant webhook fires as it would for any other outcome. A line that implies re-opening a terminal state becomes a break instead (ADR-07).

Options weighed

• ChosenBounded resolution authority through the same state machine: Closes the unknown-outcome window automatically while keeping the dangerous transitions human
• Right elsewhereBreak-only: the reconciler raises, a human resolves: Right at volumes where manual resolution is staffable, and genuinely safer; wrong at thousands a day, where the backlog becomes the failure mode
• RejectedUnbounded resolution authority: A batch job able to flip a terminal state is a batch job able to un-refund a buyer, and the blast radius of a matcher bug becomes unbounded
• RejectedDirect writes to the payment store from the matcher: Faster to build and bypasses the legality check, which is the only thing making this decision safe

Consequences

What it buys

• The unknown-outcome window closes without a human, which is what makes ADR-04's 'never retry' rule operationally viable
• Resolution reuses the ordinary transition, event and posting path, so it is exercised by the same tests and metrics as everything else
• `source = reconciliation` makes resolved payments queryable, so the rate of platform-provider disagreement per provider is a measurable number
• The dangerous transitions remain dual-control, so the authority granted is bounded by the state machine rather than by trust

What it costs

• There is a real race between the matcher and a late callback arriving for the same payment; it resolves correctly because both go through the legality check, but reasoning about it is genuinely harder
• A matcher bug can now affect payment state, not just reporting — the blast radius is bounded but not zero
• The boundary between 'pending, resolvable' and 'terminal, dual-control' is a judgement call that will need revisiting per provider
• Merchants receive a webhook up to 24 hours after a payment they were told was processing, which some integrations will handle badly

Choose differently when

If providers offered reliable inquiry APIs (see ADR-04), the matcher's resolution authority would be needed for a small residue rather than for the main path, and the safer break-only posture would become affordable. The authority exists because the automatic alternative does not.

Why it holds up over time

The durable part is the shape of the grant: authority bounded by a state machine rather than by convention, with the source recorded. That pattern — let the batch job act, but only through the same door as everything else — transfers to any system where an asynchronous process must correct a synchronous one.

Does money rest in the platform's accounts between collection and payout, or does it never stop?

Context

This is the decision that determines perhaps a third of the ledger's account model and most of the platform's regulatory posture, and it is often made implicitly. Pass-through keeps the platform thin: funds settle to the merchant's own account and refund funding and dispute liability sit with the merchant and the acquirer. Holding a balance makes refunds, partial refunds, dispute debits, fee netting, hold periods and marketplace splits all natural operations on an internal account — and makes the platform custodian of other people's money, with segregated client accounts, float accounting, a licence, and a far larger ledger.

Decision

Hold merchant balances. Compute each merchant's available balance as settled receipts minus fees, taxes, refunds, disputes and prior payouts, with a configurable hold period per risk tier, and initiate payouts on a per-merchant schedule. Accept the consequences in full: segregated client funds, a named account model including dispute reserve and receivable accounts, float accounting, and the licensing obligations of a custodian. Permit a refund that takes a balance negative and record the result as a receivable rather than rejecting the refund.

How it is realised on this stack

The account tree carries merchant payable, platform fee revenue, tax withheld, dispute reserve, settlement clearing, receivable and break accounts, all in the ledger from day one. Available balance is a projection over the ledger with the hold period applied, never a stored number. Hold periods are merchant attributes set by the risk platform, which this service reads. Client funds sit in a segregated settlement account at the bank, and the daily cash-position comparison asserts that the ledger's view of held funds matches it within tolerance.

Options weighed

• ChosenHold merchant balances as custodian: Refunds, disputes, fees and hold periods become ledger operations; the cost is licensing and a much larger account model
• Right elsewherePure pass-through to the merchant's own account: Right for a thin gateway, and much simpler: no client funds, no float, a small ledger; wrong where refunds must succeed regardless of merchant liquidity and disputes must be debitable
• RejectedHybrid: pass-through with a small retained reserve per merchant: Carries most of the custodial obligations for a fraction of the capability, which is the worst of both
• RejectedDefer the decision and build pass-through first: The account model is the hardest thing to change after billions of entries exist; deferring it is choosing pass-through without saying so

Consequences

What it buys

• A refund succeeds even when the merchant has no liquidity, so a buyer who has been charged is never caught by their merchant's cash position
• Dispute debits, fee netting and hold periods are ordinary ledger transactions rather than external arrangements
• Payout timing becomes a platform control, which is what makes risk-tiered hold periods possible at all
• The account model needed for marketplace splits in Phase 3 already exists in outline rather than needing to be retrofitted

What it costs

• The platform is a custodian of client money: segregated accounts, float accounting, licensing and a materially heavier compliance obligation
• Negative balances are real credit exposure, and the receivable account will have genuinely uncollectable amounts in it
• Dispute reserve sizing is a risk policy this architecture requires and does not set, which is a dependency on a function outside it
• The ledger is substantially larger than a pass-through design's, which drives the partitioning and archival work in ADR-08

Choose differently when

If the licence were unobtainable in the country of operation, or if the business decided the compliance burden outweighed the product value, pass-through would be the only option and roughly a third of the ledger's accounts would disappear along with the payout engine. This is the one record whose reversal would change the shape of every other view in the set.

Why it holds up over time

Custody is a business and regulatory posture, not a technical one, and it is the most stable kind of decision there is: it changes when a licence or a business model changes, not when technology does. Which is also why getting it wrong is so expensive.

Can one authenticated human change the books?

Context

Every ledger needs a manual correction path: breaks must be cleared, misapplied fees reversed, genuine disagreements settled. That path is also the most dangerous capability in the platform, and the usual control is a written policy saying two people must agree — enforced by culture, reviewed at audit, and routinely bypassed under time pressure by exactly the people the policy is aimed at. The alternative is to make the second approval a technical precondition, which cannot be skipped because the operation does not complete without it.

Decision

Enforce dual control in the path. Any privileged operation that moves money outside the normal flow — manual ledger adjustment, payout release override, merchant balance correction — is a proposal by one authenticated human and an approval by a different authenticated human, with both recorded. Sign the resulting instruction in the HSM so the approval is cryptographic evidence rather than a row somebody with database access could write. Write an immutable audit record for every read and write of payment data by a human operator, including the justifying case reference. Mask amounts and identifiers from operator views by default and require explicit, logged elevation.

How it is realised on this stack

A dual-control service holds proposals, enforces that the approver's identity differs from the proposer's, and signs the approved instruction with an HSM key before the ledger accepts it. The ledger's correction path requires a valid signature, so the control is enforced by the store rather than by the console. Workforce authentication requires phishing-resistant MFA. The operator console is masked by default; elevation emits an audit event carrying the support case reference. Audit records are append-only with seven-year retention.

Options weighed

• ChosenIn-path dual control with HSM-signed instructions: No sequence exists in which one human changes the books, and the approval is evidence rather than a claim
• Right elsewherePolicy-based dual control reviewed at audit: Right in a small team where the technical control costs more than the risk; wrong anywhere the control is load-bearing, because the bypass is always available exactly when it is most tempting
• RejectedFour-eyes in the ticketing system, single actor in the platform: The common compromise: the audit trail looks right and the technical capability is unchanged, so the control is theatre
• RejectedNo manual correction path at all: Admirably strict and unworkable: breaks and genuine errors need resolution, and removing the path means someone eventually does it with raw SQL

Consequences

What it buys

• A single compromised operator account cannot move money, which changes the value of phishing a finance team member
• The approval is cryptographic, so it survives a compromise of the application database that could forge an approval row
• Every privileged read is attributable to a case, which makes the insider-curiosity question answerable rather than merely discouraged
• The control is enforced by the ledger, so a new console or a direct API call is subject to it too

What it costs

• Clearing a break needs two people available, which is slow at 03:00 and will be complained about in exactly the incidents where speed matters
• A dual-control service on the correction path is another component to keep available, and its unavailability blocks all corrections
• Masking by default makes routine support slower and generates elevation volume that someone must review
• The control is only as strong as the separation between the approvers, which the architecture cannot enforce

Choose differently when

If the platform held no funds (ADR-13) and corrections only affected reporting rather than money owed, the HSM signing would be disproportionate and a policy control would be adequate. The cryptographic enforcement is justified by custody: the balances being corrected are other people's money.

Why it holds up over time

"Make the control a precondition rather than a rule" outlives every identity provider, HSM and console in the design. The specific mechanism will be replaced; the property that the operation does not complete without a second human will not.

When screening flags a payout, and when the books have not been proved, what happens to the money?

Context

Two gates sit in front of money leaving the platform and they fail in opposite directions. Sanctions screening produces false positives at a meaningful rate against common names; rejecting on a match would permanently strand a legitimate merchant's funds and create a dispute with no obvious resolution path, while ignoring a match is a regulatory breach. Separately, a payout computed from a ledger that has not been reconciled and proved is a payout of money the platform believes it holds, and the gap between believed and actual cash is exactly what reconciliation exists to find.

Decision

Screen payees against sanctions lists before payout and hold rather than reject on a potential match, routing the hold to a review queue with an owner and an ageing clock. Gate the payout run on the daily trial balance and the bank cash-position comparison, so payouts move proven cash rather than believed cash. Treat a payout as pending until the bank confirms, and handle a returned or failed payout by restoring the balance and recording the failure reason. Make payout instructions idempotent on a unique instruction key so a retry cannot send money twice.

How it is realised on this stack

Screening runs as a pre-payout step; a match sets the payout to `held` with a reason and raises a review item rather than failing it. The 02:00 close produces the trial balance and the cash-position comparison; a tolerance breach blocks the 06:00 payout run entirely rather than paying out partially, because a partial run on unproven books is harder to unwind than a delay. Payouts carry `UNIQUE (instruction_key)`. A bank return credits the balance back through a normal ledger transaction.

Options weighed

• ChosenHold on match; gate the run on a proved trial balance and bank comparison: False positives are recoverable, and no payout moves money the books have not proved
• RejectedReject on a sanctions match: Strands legitimate funds permanently and gives the merchant no path back, which is both a worse outcome and a worse compliance posture than a reviewed hold
• Right elsewherePay out on believed balances and reconcile afterwards: Right where payout amounts are small relative to capital and speed is the product; wrong as a custodian, where paying out money you do not hold is the failure the licence exists to prevent
• RejectedBlock only the matched merchant's payout and run the rest on unproved books: Conflates the two gates: the screening gate is per merchant, the proof gate is about the whole ledger

Consequences

What it buys

• A false positive costs a merchant a delay and a review, not their funds, which is the difference between a support conversation and a legal one
• No payout can exceed what the books have been proved to hold, so the licence obligation is enforced by the pipeline rather than by care
• An idempotent instruction key makes payout retries safe, which is what allows retrying at all
• A returned payout flows back through the ledger, so the merchant's balance history explains the round trip

What it costs

• A tolerance breach in the cash comparison blocks every payout that morning, so one unexplained discrepancy delays 50,000 merchants
• Held payouts need a staffed review queue, and an unstaffed queue turns a hold into an indefinite freeze — the outcome the decision was meant to avoid
• Merchants see a balance go up and then back down when a payout is returned, which is honest and confusing
• The tolerance threshold is a judgement call that trades false blocks against undetected drift

Choose differently when

If the close-of-books proof could be produced continuously rather than nightly, the all-or-nothing payout gate could become per-merchant and a single discrepancy would stop paying one merchant rather than all of them. That is a worthwhile future change and it depends on the ledger partitioning in ADR-08 supporting a continuous close.

Why it holds up over time

Both rules are about the direction a gate fails in: hold rather than reject where a false positive harms an innocent party, and refuse rather than proceed where the books are unproved. Those directions do not change with screening technology or accounting tooling.

How does the platform know that a balance on a dashboard is derived from the ledger rather than quietly authoritative?

Context

Projections drift into becoming truth. A balance table is created as a cache, an incremental updater is written for speed, a bug or a missed event leaves it slightly wrong, and because nobody ever rebuilds it the drift is permanent and invisible. Eventually a correction is applied to the projection rather than to the ledger, and from that moment the projection holds information the ledger does not. Claiming a store is derived is cheap; proving it requires actually deleting and rebuilding it, and comparing.

Decision

Treat every derived store — merchant balances, statements, analytics marts, dashboard caches — as rebuildable from the ledger and the event log, and prove it by rebuilding at least one projection on a schedule and comparing the result. No correction is ever applied to a projection; corrections go to the ledger and flow through. Keep settlement, reporting and analytics workloads off the stores in the payment path, and report their infrastructure cost separately so a reporting workload cannot quietly consume the payment path's budget.

How it is realised on this stack

Projection builders are batch jobs running in DC-2 reading the ledger and the event log, writing to versioned target schemas so a rebuild is a build-and-swap rather than a mutation. A scheduled rebuild of the balance projection runs weekly and diffs against the incrementally maintained copy; a non-zero diff is a paging alert, because it means either the incremental path or the rebuild is wrong. Projection stores have no write path from the operator console. Cost attribution tags separate payment-path from reporting infrastructure.

Options weighed

• ChosenRebuildable projections with a scheduled rebuild-and-compare: The claim that a store is derived is continuously tested rather than asserted
• Right elsewhereIncrementally maintained projections, trusted: Right where projections are cosmetic and a stale number costs nothing; wrong where the number is a merchant's balance and someone will eventually correct it in place
• RejectedRebuild only on demand, when a discrepancy is reported: The rebuild path then has no test coverage and will be broken exactly when it is first needed
• RejectedCompute balances on read from the ledger, no projection: Correct by construction and too slow at 35 billion entries; it also puts dashboard load directly on the payment path's store

Consequences

What it buys

• A drift between the incremental path and the ledger is caught within a week rather than at an audit
• The rebuild path is exercised routinely, so a disaster-recovery rebuild of projections is a known operation rather than an experiment
• No operator write path to projections means the tempting shortcut — fix the number the merchant can see — does not exist
• Reporting load is physically separated from the payment path, so an expensive query cannot slow a payment

What it costs

• A weekly full rebuild of a large projection is real compute, and its cost grows with ledger volume
• The rebuild-and-compare job will become slow and will be the first thing someone proposes to disable, which is how this control decays
• Build-and-swap schemas mean projection storage is roughly doubled during a rebuild
• A diff alert is ambiguous — either side could be wrong — so it requires investigation rather than a mechanical fix

Choose differently when

If projections were small enough to compute on read, none of this machinery would be needed and the ledger would serve the dashboard directly. At 35 billion entries that is not available, and the rebuild-and-compare is the cheapest substitute for the guarantee that on-read computation would have given for free.

Why it holds up over time

"Prove the derivation by performing it" is a general discipline that applies to caches, search indexes, materialised views and anything else claimed to be downstream of a source of truth. The claim decays silently; the scheduled proof does not.

Where does a regulated money path run, and what is the operational price of that answer?

Context

The default answer for a platform of this shape is a managed cloud: fewer components to operate, managed brokers and databases, and an HSM service. Three constraints push against it here. Payment data for domestic transactions must stay in the country, including backups and disaster-recovery copies, which narrows the region choice and makes every managed service's replication topology a compliance question. PCI-DSS scope containment is easier to demonstrate with a physically segmented cardholder data environment than with a shared-tenancy service. And the money path should not depend on a provider the regulator has not assessed. Against that: self-managing Kubernetes, Kafka, PostgreSQL and HSMs is a standing headcount commitment, and an understaffed platform team is the most likely cause of an outage in any design.

Decision

Run the platform on self-managed open source across two in-country data centres, 40 km apart at 1.4 ms round-trip. Both sites serve authorization traffic, with DC-2 carrying roughly 30% in steady state so failover is a weight change rather than a cold start. Replicate the ledger synchronously between them for RPO 0. Place settlement, payout and projection batch workloads in DC-2 by default. Accept that there is no third site, and that a metro-wide event is a declared, announced outage with RPO 0 rather than a failover.

How it is realised on this stack

Kubernetes with three control-plane nodes per site; Kafka with five brokers, replication factor 3 and min.insync.replicas 2, stretched across both sites; PostgreSQL primary in DC-1 with a synchronous standby in DC-2; network HSM pairs in each site's CDE segment; Envoy at the edge behind an anycast VIP. Provider egress leaves through an allowlisted proxy on static IPs over IPsec and leased lines, which is itself easier to arrange from owned infrastructure than from a cloud egress.

Options weighed

• ChosenSelf-managed open source, two in-country data centres: Residency and PCI scope are structural rather than configured; 40 km buys synchronous replication and RPO 0
• Right elsewhereManaged cloud in an in-country region: Right where the regulator accepts the provider and in-country regions with compliant replication exist — and genuinely cheaper to operate; wrong where every managed service's replication topology becomes a compliance argument
• RejectedHybrid: CDE on-premises, everything else in a managed cloud: A reasonable-sounding compromise that puts the ledger in the cloud and the keys outside it, which maximises the number of residency and latency boundaries on the money path
• RejectedThree sites for metro-event survival: The honest fix for the accepted risk, and the third site breaks synchronous replication and therefore RPO 0 — the trade was taken in favour of durability

Consequences

What it buys

• Data residency is satisfied by there being no other region to replicate into, rather than by configuration anyone could change
• PCI scope is a set of physically segmented racks, which is the easiest form of scope to demonstrate to an assessor
• Synchronous ledger replication at 1.4 ms makes RPO 0 affordable, which is the single strongest durability property in the design
• Static-IP, leased-line egress to money networks is straightforward from owned infrastructure

What it costs

• Self-managing Kubernetes, Kafka, PostgreSQL and HSMs is a standing headcount commitment, and the most likely cause of an outage in this design is an understaffed platform team
• Two sites in one metro means a metro-wide event takes both, with no failover available
• Capacity is bought ahead of demand: the 4,000-a-second sale peak must be physically present year round
• Every managed-service convenience — autoscaling groups, managed backups, a console — has to be built or done without

Choose differently when

If the regulator accepted a hyperscaler and a compliant in-country region existed with replication confined to the country, the managed option would win on operational cost alone and most of this record would be deleted. It is a regulatory decision presented as an infrastructure one, and it should be re-opened whenever that regulatory position changes.

Why it holds up over time

What should outlast the choice is the reasoning: residency, scope containment and dependency approval drove it, not a preference for owned hardware. If those three change, the conclusion should change with them — and the rest of the architecture, which names no vendor in its requirements, is deliberately portable if it does.