intermediate 3 min answer Multiple choice

A marketplace's payment provider is not down. Authorisation p99 rises from 400 ms to 25 s and 12% of authorisations time out. The continuity plan states an RTO of 30 minutes and documents a failover to a second provider that takes 20 minutes and requires a named executive's decision. What actually happens in the first hour?

business-continuitydegraded-modepaymentsdeclaration-criteriarto
Pick one
Show the full answer Hide the answer

Minute by minute, what happens

Minutes 0 to 15. Authorisation latency climbs. The provider's status page is green and its health check returns 200, because the health check is not a payment. No monitor is in a failed state, so no runbook fires. Checkout conversion falls and support tickets begin, which are both lagging signals measured in tens of minutes.

Minutes 15 to 40. The failure amplifies through resource holding. Each pending authorisation holds a checkout session, an inventory reservation and an application thread for up to 25 seconds instead of 400 ms — a 60-fold increase in hold time. At constant arrival rate, concurrent in-flight authorisations rise by the same factor, so the connection pool and the thread pool saturate even though the provider is technically answering. Client retries on timeout double the offered load onto a dependency that is already slow.

Minutes 40 to 60. Someone notices conversion. A call is convened. The 20-minute failover is discussed and not yet started, because the question "is this bad enough" has no numeric answer in the plan and the named executive wants to know whether the provider is about to recover. The RTO of 30 minutes has already been missed and the clock was never started.

What the user sees

Not an error. A spinner, then a vague failure, then a successful retry that may or may not have double-authorised. Partial success is worse than an outage for continuity purposes, because the business must later reconcile which attempts took money.

What stops it

A mechanism, not vigilance.

  • A declared degraded state with a numeric trigger. For example: authorisation success rate below 97% for 5 consecutive minutes, or p99 above 3 s for 5 minutes, declares a payment incident automatically. A continuity plan with no trigger for "slow" has no trigger at all.
  • A pre-authorised failover decision. The executive approval belongs in the plan's design review, not in the incident. Write the condition under which an on-call engineer may switch providers without asking.
  • A timeout below the holding budget. Cap authorisation at 3 s so slow becomes an error the rest of the system can shed, and the resource-holding amplification cannot occur.
  • Inventory holds that survive a payment failure for a bounded period, so the customer can retry without losing the basket.

Why the other options fail

  • Failover fires at 30 minutes. RTO is a target measured from incident declaration, not from degradation. With no declaration criterion for slowness, the clock is never started. This is the most common misreading of an RTO and the reason plans pass audit and fail in practice.
  • The second provider absorbs the overflow automatically. Only if someone built an automatic router with health-based steering, which the stem does not describe — it describes a documented manual procedure. Assuming the designed-for capability exists is how tabletop exercises produce false confidence.
  • Checkout recovers because timeouts are retried. Retries into a slow dependency are the amplifier, not the cure. Without a retry budget and jitter, background retries push the provider further into saturation and extend the event.

When this is the wrong answer

If the provider's degradation is a 3-minute blip, declaring and failing over costs more than it saves: a provider switch has its own risk, including duplicate authorisations across two providers and a reconciliation job nobody has run this quarter. The decision rule is a sustained window, not an instantaneous threshold — which is why the trigger above specifies 5 minutes rather than one bad sample.