A national payments rail routes billions of transactions across banks of varying reliability, and a bank switch times out after debiting but before confirming. How should timeouts, deemed status, reconciliation, reversals and customer messaging be designed so money is never lost or double-spent?
Show the full answer Hide the answer
The core problem
A timeout tells you nothing about whether the operation happened. The remitting bank may have debited and failed to respond, or may never have debited at all — and from the switch's position these are indistinguishable.
At the scale of a national rail — billions of transactions monthly across hundreds of participating banks with very different reliability — this is not an edge case. It is a continuous, high-volume condition, and the architecture is largely defined by how it handles it.
Deemed status
The rail cannot leave a transaction unresolved indefinitely, so it assigns a deemed state after a timeout: deemed success or deemed failure, according to defined rules per transaction type and per leg.
The choice of default is a business and regulatory decision with opposite risks:
- Deemed success risks crediting the beneficiary when the remitter was never debited — the rail is short.
- Deemed failure risks the remitter being debited with no credit — the customer is out of pocket, which is the outcome regulators care about most, and it is therefore usually the one the rules are written to avoid or to remedy quickly.
The deemed status is provisional, and reconciliation converts it to a true status.
Reconciliation as a first-class system
- Every participant reports its own view on a defined cycle, and the rail compares.
- Discrepancies are classified rather than merely counted: timeout at leg one, timeout at leg two, duplicate delivery, ordering, clock skew, participant defect, genuine mismatch. Each class has a different remedy, and a count without a classification is a number nobody can act on.
- Automated resolution for the classes that permit it, with human review only for the residue.
- Reversal as a defined transaction type, not an ad hoc correction — auditable, idempotent, and traceable to the original.
- Time-bounded settlement, so a customer's money is returned within a stated window rather than when someone gets to it.
- Reconciliation is a product surface, not a back-office function, because its output is customer money and its timeliness is a regulatory obligation.
The properties the design depends on
- Idempotency end to end. Every leg carries a unique transaction reference, and a retry with the same reference must return the original outcome rather than performing the operation again. Without this, retrying a timeout is how double-spending occurs.
- A durable record before any external call. The intent is written before the request leaves, so a crash mid-flight leaves a record to reconcile rather than nothing.
- Status query as a first-class API. After a timeout, the correct action is to ask — a supported operation that returns the authoritative status of a reference — rather than to retry.
- Immutable transaction history, so the sequence that produced any state is reconstructible for a dispute or an audit.
Customer messaging, which is where trust is decided
The honest message differs from the internal state, and both must be correct.
- Never say "failed" when the status is deemed — because if the debit occurred, the customer sees money gone and a failure message, and that is the interaction that destroys confidence in a payment system.
- "We are confirming this transaction; if the amount was debited it will be returned by [time]" is accurate, actionable, and sets a bound.
- Proactive notification when reconciliation resolves it, rather than requiring the customer to check.
- A stated maximum resolution time, which is frequently a regulatory requirement and is in any case the commitment that makes the uncertainty tolerable.
The architectural summary: in a payments rail, correctness matters more than latency, and the system is designed around the assumption that a meaningful fraction of transactions will end in an uncertain state that must be resolved afterwards — which is the opposite of the assumption most distributed systems are built on.