A client that times out on a mutating call holds no evidence about whether it happened, and the only safe recovery is to send it again. This guide reconstructs, from Stripe, Airbnb, Shopify, AWS, Kafka and three decades of RPC research, the machinery that makes the resend harmless: what the idempotency key actually promises, the five clauses of that contract, and the published incidents where one missing clause charged real credit cards up to fifty times.
The problem, stated without naming a technology: when a mutating request dies without a reply, resending is mandatory and re-executing is forbidden, and something has to absorb the difference.
On a night in July 2013, Twilio's billing system read every customer balance as zero, decided every account needed topping up, charged the stored credit card, failed to record that it had done so, and concluded from the failure that it should try again. It kept concluding that until on-call engineers shut the whole billing system down at 4:10 a.m. Pacific time. Twilio's postmortem reports that the auto-recharge service "billed the customers before updating their balance internally" and "continued to retry the transaction again and again, resulting in multiple charges to customer's credit cards." Nothing in that loop was exotic. Every part of it, a read, a charge, a write, a retry, is code most teams have shipped.
The underlying problem is older than every company in this guide and does not mention HTTP. A sender that gets no reply cannot distinguish "the request never arrived" from "it was applied and the reply was lost." Pat Helland's 2012 ACM Queue article Idempotence Is Not a Medical Condition states the consequence flatly: messages get lost and are retried after a timeout, so "every message may be retried and, hence, must be idempotent." The retry is not a design choice you can decline. It is a property of the substrate. The only real choice is whether the second arrival of the same intent is absorbed or executed.
The mechanism the industry converged on is a client-generated unique identifier for the
intent, sent with the request and remembered by the server. Birrell and Nelson
published it for RPC in 1984: a call identifier of machine id, process id and sequence
number that "allows elimination of duplicate call packets." Amazon shipped it for EC2
instance launches as ClientToken in 2010. Stripe's Idempotency-Key
header entered its public documentation around 2015, a dating that
Hatchet's history hunt pinned
down in 2026 after finding that most secondary accounts cite it earlier without primary
sources. Kafka rebuilt it inside the broker in 2016 as producer ids plus per-partition
sequence numbers, which KIP-98
describes as working "in a way similar to TCP." Different decades, different layers, one
mechanism.
What makes the topic worth a field guide rather than a tutorial is the gap between how
simple the idea sounds and how much of it is contract rather than code. The IETF draft that
would standardise the header, draft-ietf-httpapi-idempotency-key-header,
lists nine organisations using the Idempotency-Key header and eleven more
implementing the same concept under other names: PayPal's PayPal-Request-Id,
Square's body-level idempotency_key, OpenBanking's x-idempotency-key,
Google's requestId. Twenty implementations, at least nine spellings, and still
no RFC. Mark Nottingham's first attempt to standardise safe POST retry,
POST Once Exactly,
expired in September 2005; the current draft, which credits it, reached draft-07 in October
2025 and remains a draft. Two decades of standards effort have not caught up with a pattern
every payments company reimplements privately, which tells you where the actual knowledge
lives: in engineering blogs, incident reports and source code, not in a spec.
The machinery that guarantees "at most once" is itself software, and it fails like
software. Kafka's KIP-679
made idempotence the producer default in release 3.0.0. KAFKA-13598
then found that in 3.0.0 and 3.1.0 the config reported enable.idempotence=true
while the validation logic never actually applied it. For two releases the strongest
delivery guarantee was on in the docs and off on the wire, and nothing failed loudly,
because the absence of deduplication is silent by construction. Every failure in section
04 shares that shape: the dedupe layer breaks quietly, and the symptom appears somewhere
else, on someone's bank statement.
Scope. This guide covers request-level duplicate suppression: idempotency keys and client tokens at API boundaries, broker-level producer dedupe, and the store and lifecycle behind them. It deliberately does not cover keeping a database write and its outbound event in agreement (this practice's earlier dig on the outbox and change data capture covers that seam), saga-style business compensation, distributed transactions and two-phase commit, or the fencing and transaction internals of stream processors beyond what the dedupe layer needs.
Across Stripe's published design, Airbnb's Orpheus, Shopify's payment service, AWS client tokens and RAMCloud's RIFL, the same five responsibilities appear. Companies differ on who drives recovery and what a reused key returns, not on the shape.
The clearest public account of the server side is Brandur Leach's 2017 article Implementing Stripe-like Idempotency Keys in Postgres, written while he worked at Stripe, and it starts from a distinction the tutorials skip: identify every foreign state mutation, meaning any effect outside your transactional store. His warning generalises beyond payment calls: "It's tempting to treat emitting records to Kafka as part of atomic operations because they have such a high success rate that they feel like they are. They're not, and should be treated like any other fallible foreign state mutation." Everything between two foreign mutations can be grouped into an atomic phase and committed together; the foreign mutations themselves cannot be rolled back, only recorded and resumed past. Airbnb's Orpheus framework is the same decomposition with different names: every request splits into Pre-RPC, RPC and Post-RPC phases, where the RPC is the foreign mutation. RIFL, the SOSP 2015 exactly-once RPC layer, factors the identical problem into four sub-problems: RPC identification, completion-record durability, retry rendezvous and garbage collection. Three vocabularies, one architecture.
1. Identity of intent. The client, not the server, names the operation,
because only the client knows that two transmissions are one intent. Every implementation
scopes the key to the caller: Brandur's schema makes the unique index
(user_id, idempotency_key), and the IETF draft's security section recommends a
composite lookup key for the same reason, so one tenant cannot collide with or probe
another's keys. The client side of the clause is just as load-bearing and lives in the
SDKs: stripe-ruby
generates a key automatically the moment retries are enabled, with the comment "It is only
safe to retry network failures on post and delete requests if we add an Idempotency-Key
header," and GoCardless's client libraries do the same. A retry policy without a key is not
a resilience feature; it is a duplicate generator with jitter.
2. Atomic claim. Two copies of the same request will race, so the first
touch of the key must be a single atomic operation. Brandur's implementation wraps the
upsert in a SERIALIZABLE transaction and lets Postgres abort one of two
concurrent claimants; Shopify's payment service takes "a lock around the API call based on
the client and idempotency key" and answers the loser with a 409, per its 2023
resilient-payments
write-up. The IETF draft encodes the same split: a retry after completion gets the
stored result, a retry while the original is still running gets a conflict.
3. Parameter binding. A key must be bound to the payload it first travelled with, or a buggy client that reuses keys across different requests silently gets wrong answers replayed. Stripe errors on reuse with different parameters; AWS returns a validation error "indicating a parameter mismatch between idempotent requests," per the Amazon Builders' Library; the IETF draft says the key "MUST NOT be reused with another request with a different request payload" and sketches the fingerprint mechanism, a server-side checksum of the payload. Notably, what the fingerprint is actually for is still being argued in the working group's own tracker (issue #36), fourteen years after Stripe shipped it.
4. Progress, not just presence. The naive implementation stores "seen"
and gives you at-most-once: a crash after the foreign mutation but before the response is
stored leaves a key that can neither complete nor replay. The production implementations all
store where the request got to. Brandur's recovery points (started,
ride_created, charge_created, finished) let a retry
resume mid-lifecycle instead of restarting; Orpheus classifies every error as retryable or
non-retryable and lets clients "repeatedly fire the same request" until the state machine
drains, a strategy Airbnb calls write repair; RIFL's completion record is the same idea
durably logged with the RPC's result. Who drives that recovery is a real divergence point:
Airbnb leans on clients to keep firing, while Brandur's design adds a server-side
completer because "there can be cases where a client starts working, never quite
finishes, and drops forever." If your clients are third parties you do not control, the
completer is not optional.
5. Replay, expiry, and the reaper. A finished key stores the entire response and replays it, and at Stripe that includes failures: a comment in stripe-ruby notes that the client does not bother retrying most 500s because "our idempotency framework would typically replay it anyway." Errors are results too; replaying them is what makes the outcome stable. The store is then garbage-collected, and the window is part of the public contract: Stripe prunes keys after 24 hours and states plainly that "we generate a new request if a key is reused after the original is pruned." Expiry is the clause everyone forgets: after the window, the same key is not a retry any more, it is a fresh command.
Brandur: a table in the service's own Postgres, so claims share transactions with business writes. Airbnb: a dedicated idempotency database, sharded by idempotency key when one primary stopped scaling. Kafka: inside the broker, keyed by producer id and partition sequence. The store is on the hot path of every mutating request; Shopify measured a 50% INSERT-time reduction switching key format from UUIDv4 to ULID (2023).
Sources: brandur.org, Airbnb, Shopify
Sidekiq's own wiki is blunt: jobs run at least once, a completed job can re-run if Redis drops the acknowledgement, and "Sidekiq makes no exactly-once guarantee at all." Uniqueness is a paid Enterprise feature with an explicit time window. The job queue's refusal is the correct end-to-end reading: the platform can dedupe its own hop, never your handler's side effects.
Source: Sidekiq wiki
Brandur's reference implementation passes its own derived key
(rocket-rides-atomic-#{key.id}) to Stripe, so a crashed server plus a
client retry still cannot double-charge: the downstream provider dedupes on its copy.
Uber's platform docs describe the same assumption from the other side; external payment
providers "are normally implementing their services as idempotent message processors."
Each hop protects the hop below it by forwarding identity, not by hoping.
Sources: brandur.org, Uber, 2018
Four forks where published systems went different ways for stated reasons, and the condition that flips each one.
idempotent_creation_conflict, pointing at the already-created resource.| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Reused completed key | Replay stored response, errors included | Recompute | Recomputing re-executes side effects; replay is the definition of the guarantee | Stripe docs, stripe-ruby |
| Reused key, new payload | Validation error (422 / mismatch) | Serve the cached response | Serving would hand caller B caller A's result | IETF draft, AWS Builders' Library |
| Concurrent duplicate | 409, caller retries shortly | Block until first finishes | Blocking ties up a worker for the length of a foreign call | brandur.org, Shopify |
| Claim concurrency control | SERIALIZABLE upsert (Brandur) or explicit lock + 409 (Shopify) | Read-then-write | Two racing claimants both pass the read | brandur.org |
| Key format | ULID | UUIDv4 | Timestamp-prefixed keys keep the index hot at the right end: 50% faster INSERTs at Shopify | Shopify, 2023 |
| Who generates keys | SDK auto-generates when retries are on | Trust integrators to remember | The unsafe combination (retry without key) should be unrepresentable | stripe-ruby, GoCardless |
| Broker default | Idempotence on (Kafka 3.0+) | Opt-in (2017–2021) | KIP-679's analysis found acks=all costs little; took a second KIP and four years | KIP-679 |
Five published incidents and one confirmed contract break, in three classes: the retry loop with no dedupe, the dedupe machinery itself failing, and duplicates born outside the boundary your keys can reach.
The public record here is thin in an instructive way. Trawling the maintained postmortem collections, Twilio 2013 is the only classic write-up whose mechanism is a duplicate-side-effect retry loop; most duplicate-charge events surface as status-page notices and press coverage rather than engineering postmortems. Two readings are possible: the pattern mostly works, or duplicate side effects get classed as billing errors and settled by support teams without an engineering write-up. The Zuora and Coinbase entries below, both documented outside engineering blogs, point to the second reading. Treat the scarcity as a detection gap, not as evidence of safety.
What the dedupe layer costs, how long the promises last, and what the incidents measured, each figure with its context and date.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Exactly-once RPC overhead | <4% on 13.5 µs writes | RAMCloud (RIFL) | Measured; durable completion records in a research system; distributed transactions ~20 µs | 2015 | SOSP paper |
| Key-insert cost reduction, ULID vs UUIDv4 | −50% INSERT duration | Shopify | Measured; one high-throughput payment system's idempotency-key table | 2023 | Shopify Engineering |
| Dedupe window | 10 minutes | DynamoDB TransactWriteItems | Vendor-documented; same token after the window is a new request | current docs, 2026 | AWS docs |
| Dedupe window | 24 hours (pruning eligible) | Stripe | Vendor-documented; "we generate a new request if a key is reused after the original is pruned" | current docs, 2026 | Stripe API reference |
| Dedupe window | ≥7 days | Adyen | Vendor-documented minimum validity after first submission | current docs, 2026 | Adyen docs |
| Broker idempotence default cost | "not significant" | Apache Kafka | Claimed in KIP-679's analysis of acks=1 to acks=all; the KIP made it default in 3.0 | 2020 | KIP-679 |
| Releases with the default silently off | 2 (3.0.0, 3.1.0) | Apache Kafka | Reported; config validation bug, fixed in 3.0.1/3.1.1/3.2.0 | 2022 | KAFKA-13598 |
| Duplicate-event exposure window | ~22 hours | CircleCI | Derived: duplicate workflows/emails Dec 2 16:20 to Dec 3 14:20 per the incident report | 2025 | incident report |
| Max duplicate charges on one card | up to 50 | Coinbase customers | Reported in press; reversal-and-repost inside the card network | 2018 | TechCrunch |
| Implementations catalogued by the would-be standard | 20 orgs, ≥9 spellings | IETF HTTPAPI WG | Reported in the draft's implementation-status section | 2025-10 | draft-07 |
| Recommended reaper horizon | ~72 hours | Stripe pattern (Brandur) | Stated rationale: survive a bad Friday deploy through to a Monday fix and completer replay | 2017 | brandur.org |
Measured: the RIFL and Shopify rows. Vendor-documented promises: the window rows, which are contracts that can change (the EC2 entry in section 04 is exactly such a change). Claimed: KIP-679's cost analysis, which the community accepted but which your workload should re-measure. Derived: the CircleCI window, computed from the report's own timestamps. Unknown, because nobody publishes it: production duplicate-arrival rates, key-store hit ratios, and the p99 cost of the key check on the request path. If you deploy this machinery, you will have to generate those numbers yourself; rung 7 below is designed to.
Every source behind this page, graded. The full claim-by-claim ledger ships beside this file as sources.md. Filter by kind.
The canonical duplicate-charge postmortem: read-only balance store, charge-then-record ordering, and a retry loop that kept charging until humans intervened at 4:10 a.m.
A race condition published duplicate workflow-completion events; the auto-rerun consumer acted on each one, creating duplicate workflows and cascading load.
A gateway introduced response code 136; the unmapped code was handled as a retryable failure and "may have resulted in duplicate charges" until the mapping was fixed.
"This issue was not caused by Coinbase": a merchant-category change made the card network reverse and repost transactions, and cardholders saw the reposts first.
The client half of the contract in production code: "It is only safe to retry network failures on post and delete requests if we add an Idempotency-Key header," plus the note that Stripe's idempotency framework replays stored 500s.
The config said true; the validator never applied it. Two releases of the strongest delivery guarantee existing only in documentation.
Replay-after-terminate behaviour documented since 2010 changed underneath users; confirmed as a bug. The provider's dedupe contract moved.
The job tier's deliberate refusal: completed jobs can re-run if the acknowledgement is lost, so handlers must be idempotent; uniqueness is a paid feature with a declared window.
The payload-binding mechanism, shipped by Stripe since the mid-2010s, is still an open argument in the standards tracker, including an internal consistency question about the draft's own sections.
The would-be standard, fetched in full: uniqueness and no-reuse-across-payloads as MUSTs, replay for completed retries, conflict for concurrent ones, and an implementation-status section listing twenty organisations.
The first standardisation attempt: a resource that answers POST successfully exactly once. Expired September 2005; the current draft explicitly credits it.
KIP-98 designs broker-side dedupe as producer id plus per-partition sequence numbers, "similar to TCP," and weighs the message-format overhead. KIP-679 makes it the default four years later after judging the cost negligible.
The most complete public server-side design: foreign state mutations, atomic phases, recovery points, the completer and the reaper, with a runnable reference implementation.
Orpheus: Pre-RPC / RPC / Post-RPC phases, retryable versus non-retryable error classification, client-driven write repair, and a dedupe database sharded by idempotency key when one primary stopped scaling.
The operational numbers nobody else publishes: ULID keys halving INSERT time, windows of 24 hours or less, and lock-plus-409 on concurrent reuse.
The client contract from the provider's mouth: generate a unique ID per operation, retry with the same ID, back off with jitter.
The deliberate dissenter: one successful use per key, then 409
idempotent_creation_conflict, with client libraries auto-generating keys.
Design goals stated as "exactly-once payment processing by the means of idempotency and strong consistency," implemented through immutable orders persisted before processing and a versioned entity change log, with double-entry bookkeeping as the audit layer that catches what dedupe misses.
A primary-source hunt through the pattern's history: Stripe's header dates to ~2015 in the documented record, EC2's ClientToken precedes it, and the trail ends at PARC-era RPC research.
The argument from first principles that retries are a property of loosely coupled systems, so every message must tolerate re-arrival.
Exactly-once as a reusable infrastructure layer: RPC identification, durable completion records, retry rendezvous, garbage collection; under 4% overhead on microsecond-scale writes.
The origin: call identifiers of machine id, process and sequence number that allow "elimination of duplicate call packets" so a procedure is not executed twice when only the acknowledgement was lost.
Two years after shipping, the implementer's accounting of "remaining gaps" in exactly-once and how they were being addressed; slides published alongside.
The mechanism decomposed for practitioners: idempotent producer, transactions and fencing as three separable pieces rather than one magic property.
Independent reporting quantifying the blast radius of network-level duplicates: cards charged 17 and 50 times, drained accounts, overdraft fees.
Amazon's institutional pattern: client tokens on mutating APIs, parameter-mismatch validation errors, and the reasoning a principal engineer gives internal teams.
Three current vendor documents defining how long the same key means "the same request": Stripe prunes at 24 hours, Adyen holds at least 7 days, DynamoDB transactions 10 minutes. Same word, three orders of magnitude.
The same contract transported in the request body rather than a header, with replay of the stored success and an error on reuse with changed parameters.
Seven rungs from an evening's middleware to the operational surface. The toy-to-real line is crossed at rung 4, where you start killing the process on purpose.
A toy API with one mutating endpoint; a map from key to stored response; reject requests missing the header.
Done when: two identical curls return byte-identical responses and the side effect count is 1. Teaches: replay is the guarantee, not "we noticed the duplicate."
Unique index on (caller, key); store a payload fingerprint and the response; return a validation error on fingerprint mismatch and 409 while the first attempt holds the lock, per the IETF draft's enforcement table.
Done when: two parallel requests with one key produce exactly one 2xx and one 409, under a loop of 1,000 runs. Teaches: the atomic claim, and why read-then-write cannot provide it.
Add a fake payment provider with its own key support and nonzero latency. Split the endpoint into atomic phases with recovery points, forwarding a derived key downstream, following the Stripe pattern.
Done when: the state machine's phases match Figure 3 and every phase commits before the next foreign call. Teaches: Brandur's rule that the foreign mutation is the unit the design bends around.
kill -9 between the provider call and the response write; retry with the same key and watch it resume from the recovery point instead of double-charging.
Done when: 100 randomized kill points yield zero duplicate charges and zero stuck requests. Teaches: progress-not-presence; the naive "seen" flag fails this rung.
A background process that finds unfinished keys whose clients gave up and drives them to completion; a reaper that prunes finished keys past the window. Then write the test that reuses a key after pruning and observe the duplicate it creates.
Done when: the expiry-duplicate test passes only when the client's maximum retry horizon is shorter than the window. Teaches: the window is a contract clause, and Stripe's docs state its failure mode outright.
Reproduce Zuora's bug (make the provider return an unknown response code; assert it parks the attempt instead of retrying) and Twilio's (make the local record store read-only after the charge; assert the loop halts and alarms).
Done when: both faults produce a paged alert and zero additional provider calls. Teaches: the three-outcome model, and that the dedupe layer needs its own failure detection.
Load-test the key check's p99 on the request path, the store's growth rate against the reaper, and duplicate-arrival rate under injected client retries; compare ULID and UUIDv4 key formats against Shopify's 50% figure. Add a reconciliation query that counts provider-side charges per intent, the control Uber's double-entry design and Twilio's independent record both point at.
Done when: a dashboard shows duplicate-absorption rate, key-store p99 and reconciliation drift, and an on-call runbook says what to do when each moves. Teaches: day-two ownership; prevention plus detection, never one alone.
The queries that found this material, grouped by what they surface. The domain vocabulary that unlocks the good layer: "double charge," "client token," "exactly-once," "at-least-once," "write repair," "response code mapping."
"idempotency" payments postmortem "double charge" incident retrystatus page incident "duplicate charges" OR "duplicate webhooks" retry root causebilling incident post-mortem redis "charged multiple times"site:github.com danluu post-mortems billing duplicate"idempotency keys" postgres implementing stripe-like brandurairbnb "avoiding double payments" idempotency orpheusshopify engineering "resilient payment systems" idempotency ULIDuber engineering payments idempotency "exactly once" money movementIETF idempotency-key header draft httpapi working group"POST once exactly" Nottingham draft expiredKIP-679 idempotence default kafka "acks=all"KAFKA-13598 enable.idempotence default not appliedboto3 issue RunInstances "not idempotent anymore" client tokenBirrell Nelson "implementing remote procedure calls" 1984 duplicate callRIFL "implementing linearizability at large scale" SOSP exactly-once RPCHelland "idempotence is not a medical condition"EC2 RunInstances client token idempotency window docs