A card-processing platform's downstream authorisation network is slow but not failing - responses arrive, just far too late. Why does a standard error-rate circuit breaker not help, and what does?

Why the error-rate breaker misses it

A breaker that trips on error rate sees success. Every call eventually returns 200. The error rate is near zero, so the breaker stays closed while every request holds a connection and a thread for several seconds.

Slow is worse than down. A dependency that fails fast returns your resources immediately. A dependency that takes eight seconds to succeed consumes them for eight seconds each, and at any meaningful request rate that exhausts the pool. The service then fails for reasons unrelated to the slow dependency — new requests cannot get a connection — which makes diagnosis hard because the symptom appears everywhere except the actual cause.

What actually works

• Trip on latency, not only on errors. If p99 to this dependency exceeds a threshold for a sustained window, open the breaker. A slow dependency is a failing dependency for capacity purposes.
• A concurrency limit per dependency, which is the more fundamental control. A semaphore capping in-flight calls to a specific downstream means that no matter how slow it becomes, it can only ever consume its allocation. This is the bulkhead, and it is more reliable than the breaker because it needs no threshold tuning and no detection delay — it is a hard bound rather than a reaction.
• Aggressive timeouts derived from the deadline, so a call that cannot possibly be useful is abandoned rather than waited on.
• A separate pool per downstream, so exhaustion is contained to the callers of that specific dependency.

The specific danger in card authorisation

An abandoned authorisation is ambiguous, not failed. Timing out does not mean the network did not authorise — it may have, and the cardholder's balance may be held. So the timeout must lead to a defined resolution path: a reversal, a status query, or a pending state that reconciliation will close.

A circuit breaker that opens on a payment path must not simply return an error to the caller if requests may already have taken effect downstream. It must return a state that the rest of the system knows how to resolve. This is the interaction between resilience patterns and financial correctness that gets designed last and matters most.

The rehearsal

Breakers and bulkheads that have never been exercised do not work. The thresholds are guesses until a fault injection exercise — deliberately adding latency to a dependency in a controlled window — shows what the service actually does. A breaker tested only in unit tests has been tested against a mock, not against saturation.