A digital bank needs automatic failover for availability, but some operations cannot safely execute twice. Which components fail over automatically, which degrade, and which stop?

The three categories

Fail over automatically: stateless services, read paths served from replicas, caches, edge routing, notification delivery. Loss of a region costs capacity rather than correctness, and automatic recovery is strictly better than waiting for a human.

Degrade automatically: anything that can serve stale data usefully. Balance display from a replica with an as-of timestamp, transaction history, statements, spending analysis. The product must be able to express "as of a few minutes ago", which is a design requirement rather than an error state.

Stop, and require a decision: the authoritative write path for money. Payment initiation, transfers, withdrawals. Automatic promotion of a strongly consistent primary risks two primaries, and a split ledger is far more expensive than an hour of unavailability.

Why stopping is the correct engineering answer

The instinct is that stopping is a failure of design. In a financial system it is the design. Reconciling two divergent write histories for customer balances is not a technical exercise — it involves regulators, customers, and money that has already moved. The expected cost of that vastly exceeds the expected cost of downtime, and the arithmetic is not close.

The mitigation is not automatic failover but reducing the scope of what stops: entity homing, so a regional failure stops writes for a subset of accounts rather than all of them.

What makes stopping survivable

• Read paths that keep working, so most customers see their money even though they cannot move it.
• A clear customer message distinguishing "temporarily unable to transfer" from a generic error, because the second generates support load and panic.
• Queued intent where the domain permits it — accepting an instruction for later execution, clearly labelled as pending — which converts an outage into a delay for the operations that tolerate it.
• A promotion procedure that is fast once the decision is made, so the human decision is the only delay rather than the decision plus an hour of manual steps.

The rehearsal

The decision is usually slower than the failover. Teams rehearse the technical promotion and not the judgement: who declares the region gone, on what evidence, and what happens if they are wrong. That delay is what consumes the recovery time objective, and it is the part that is never practised.