Integration & APIs 01 Sep 2026 33 min read

A timeout is not an answer: making retried requests safe

How production systems make a resent request harmless: idempotency keys, client tokens and dedupe stores, what the guarantee actually covers, and how the machinery itself fails.

A client that times out on a mutating call holds no evidence about whether it happened, so it must resend, and something has to make the resend harmless. This guide reconstructs that machinery from Stripe's published design, Airbnb's Orpheus, Shopify's payment service, AWS client tokens, Kafka's idempotent producer and four decades of RPC research, then walks five published incidents where one missing clause of the contract charged real cards up to fifty times. After reading it an architect can name the five clauses their own endpoint needs, pick replay versus conflict and a dedupe window deliberately, and defend both in a design review.

The finding that surprised me

The machinery that guarantees at-most-once fails silently like any other software: Kafka 3.0.0 and 3.1.0 shipped idempotence 'on by default' while a validation bug left it off, and nothing failed loudly, because the absence of deduplication produces no error, only occasional duplicates.

What you get out of it

  • The idempotency key is RPC's 1984 call identifier rebuilt at a new layer roughly once a decade, and twenty organisations now run it under at least nine different header spellings because two decades of standardisation attempts (POE 2005, the IETF draft since 2021) have not landed.
  • The contract has five clauses, not one: client-generated identity, atomic claim, payload binding, recorded progress (not just presence), and replay with a published expiry window; every published incident maps to one missing clause.
  • The dedupe window is part of the public contract and spans three orders of magnitude across vendors (DynamoDB 10 minutes, Stripe 24 hours, Adyen 7 days), and Stripe's own docs state the failure mode: a key reused after pruning starts a new request.
  • Platform dedupe covers only the hop the platform owns: Sidekiq refuses to promise exactly-once, and CircleCI's December 2025 incident shows duplicate events cascading through a non-idempotent consumer despite everything upstream.
  • Duplicates can be born below every layer you operate, as in the 2018 Visa/Worldpay reversal-and-repost event, so prevention needs a paired detection control: reconciliation against provider records, which is what saved Twilio in 2013.

Scope

Why this, now. The IETF draft that would finally standardise the Idempotency-Key header reached draft-07 in October 2025 and is still not an RFC, while the freshest incident in the catalogue (CircleCI's duplicate-event cascade) is from December 2025, so the pattern is simultaneously forty years old and still going wrong in production this year.

What it does not cover. Keeping a database write and its outbound event in agreement (covered by this collection's outbox and change-data-capture dig), saga-style business compensation, distributed transactions and two-phase commit, and stream-processor transaction and fencing internals beyond the dedupe layer.

Open the field guide → Self-contained: it loads nothing at read time, follows your system theme, and prints cleanly.