A financial-data aggregator depends on hundreds of external banking APIs, some slow, some intermittently down, all with different behaviour. How should timeouts, retries, circuit breakers, caching, stale data and provider isolation interact?

The core insight

One retry and timeout policy across hundreds of heterogeneous providers is guaranteed to be wrong for nearly all of them. A bank that normally answers in 300ms and one that normally takes 8 seconds cannot share a timeout: set it for the fast one and you fail the slow one permanently; set it for the slow one and you hold connections for 8 seconds against a provider that died in 300ms.

Per-provider policy, derived from observation

• Timeout set from the provider's own observed latency distribution — typically just above its p99 — and recalculated automatically rather than configured once and forgotten.
• A concurrency limit per provider, which is the actual isolation mechanism. The failure you are preventing is one slow provider consuming every worker in the pool, which turns a single bank's bad afternoon into a total platform outage. A per-provider semaphore caps the damage at that provider's share.
• A circuit breaker per provider, opening on error rate and on latency, because a provider that is slow but not failing is more dangerous than one that is down — it consumes resources for the full timeout and returns nothing.
• Retry budgets rather than retry counts. A per-provider budget expressed as a fraction of total requests (say 10%) makes retry load bounded during a broad failure, which a per-request count does not.

Where caching and staleness come in

For aggregated financial data, stale is usually acceptable and unavailability is not. The design should serve last-known-good with an explicit as_of timestamp and let the consuming application decide. That single choice removes most of the availability pressure from the provider integration, because a provider outage degrades freshness rather than function.

The exception is any flow where the customer is about to act on the number — a payment initiation, a balance check before a transfer. Those need a live read or an explicit refusal, never a silently stale value.

The deadline that ties it together

Propagate a deadline from the inbound request rather than configuring timeouts per hop. If the caller has 2 seconds and 1.4 have been spent, the provider call gets 600ms and the retry does not happen. Independent per-hop timeouts sum to something far larger than any caller will wait for, so work continues on requests nobody is listening to any more — which is pure waste at exactly the moment capacity is scarce.

Reconciliation

None of this makes the data right. A background reconciliation that re-fetches and compares is the only control that detects a provider returning stale or wrong data successfully, which is the failure mode that none of the resilience patterns above can see.