A payments platform sees a 20x increase in transaction attempts during a major commerce event. Idempotency, rate limiting, queueing, fraud checks, database contention and a downstream provider all interact. What is the correct ordering of these controls on the request path, and why does ordering matter more than any single control?

Why ordering is the whole answer

Every one of these controls is cheap in isolation and each protects a different resource. Put them in the wrong order and you spend the expensive resource before deciding whether to do the work at all. The rule is: reject early, deduplicate before doing work, and hold the durable write as late as it can safely be held.

The ordering that survives

1. Edge rate limiting, per merchant and per credential. Costs a counter increment. It exists to stop a single caller's retry storm from consuming capacity everybody else needs — and during a surge, retry storms from mobile clients are a larger share of the load than genuine new attempts.
2. Idempotency lookup, before anything else touches the payment. A repeated key returns the stored response. This must sit ahead of fraud checks and provider calls, or a client retry costs a fraud evaluation and possibly a second authorisation.
3. Admission control on the payment-initiation path — a bounded queue with a fast rejection when full. An unbounded queue during a 20x surge converts a capacity problem into a memory exhaustion problem, and then into a total outage.
4. Durable persist of the attempt with its idempotency key, in one transaction, before calling the provider. This is the point of no return, and it must happen before the external call so that a crash mid-call leaves a record to reconcile from rather than an invisible payment.
5. Fraud evaluation with a deadline and a defined behaviour on timeout. Under surge, the fraud service is itself under load; a synchronous fraud check with no deadline is the most common way a payment path collapses.
6. The provider call, with per-provider bulkheads, a circuit breaker, and a retry policy that only retries what is safe to retry.

The interactions that actually cause duplicates

• Retry without idempotency at any layer — client, gateway, or your own retry loop. Each one multiplies.
• An idempotency key scoped too narrowly, so the same logical payment gets two keys and both succeed.
• Writing the attempt after the provider call, which loses the record of a payment that succeeded downstream and is unknown upstream. This single ordering choice is the origin of most reconciliation breaks.
• A circuit breaker opening on a provider that is slow rather than failing, converting timeouts into ambiguous outcomes with no terminal state.

The thing teams forget

Ambiguity is the normal case, not the exception. A provider timeout does not mean the payment failed; it means you do not know. The architecture must have a place to put "unknown" — a pending state, a reconciliation job, a status query against the provider — because a system that has only success and failure will guess, and guessing about money is how duplicates are born.