Idempotency in Practice: Why Your Retry Just Charged the Customer Twice

The retry is not the bug. The retry is correct behaviour — your client library did exactly what it was configured to do when the connection dropped.

The bug is that your server had no way to know it had seen that request before.

This is the most consequential thing in API design that almost nobody gets fully right, and it has a name that everybody recognises and surprisingly few can define precisely: idempotency.

The Definition That Actually Matters

An operation is idempotent if performing it many times has the same effect as performing it once.

RFC 9110 — the current HTTP semantics standard — is careful to phrase this in terms of intent:

“A request method is considered ‘idempotent’ if the intended effect on the server of multiple identical requests with that method is the same as the effect for a single such request.”

That word intended does a lot of work. The spec immediately clarifies that a server is free to log each request separately, keep a revision history, or have other non-idempotent side effects. Idempotency is about what the client asked for, not about whether anything anywhere changed.

Which methods qualify?

Safe and idempotent are different properties. Every safe method is idempotent; the reverse is false. DELETE is idempotent and absolutely not safe.

One caveat on that table: PATCH is not defined by RFC 9110 at all. It comes from RFC 5789, which says outright that “PATCH is neither safe nor idempotent” — while adding a detail that previews this article’s whole argument: “A PATCH request can be issued in such a way as to be idempotent.” The method doesn’t decide. The implementation does.

Note what’s missing from the “yes” column: POST. That single fact is the reason this entire article exists.

The Spec Tells You Not To Retry. Everyone Retries Anyway.

RFC 9110 is explicit about why idempotency was defined at all:

“Idempotent methods are distinguished because the request can be repeated automatically if a communication failure occurs before the client is able to read the server’s response.”

And then it draws the line:

“A client SHOULD NOT automatically retry a request with a non-idempotent method unless it has some means to know that the request semantics are actually idempotent… A proxy MUST NOT automatically retry non-idempotent requests.”

Now look at reality. Service meshes retry. Load balancers retry. HTTP client libraries retry by default. Mobile apps retry when the user taps twice because the spinner looked stuck. Message queues redeliver.

The spec’s advice is correct and completely unenforceable. So the practical question is not “how do I stop retries?” — it’s “how do I make my POST endpoint safe to retry?”

That is what the Idempotency-Key header is for.

Idempotency-Key Is a De-Facto Standard, Not a Standard

Worth knowing before you build on it: there is an IETF draft — draft-ietf-httpapi-idempotency-key-header, in the httpapi working group — and it has expired without becoming an RFC.

So when someone tells you “it’s the standard header”, the accurate version is: it’s a convention that Stripe popularised, that most payment and API platforms copied, and that has no normative document behind it. The header name is consistent across the industry. The semantics are whatever each provider decided.

That means retention windows, parameter-mismatch behaviour, and concurrency handling all differ between providers. Read the docs of whatever you integrate with — do not assume.

How the Stripe Model Actually Works

Stripe’s implementation is the closest thing to a reference, and its details are more thoughtful than most copies of it.

curl https://api.stripe.com/v1/customers \
  -H "Idempotency-Key: 8f14e45f-ea8d-4b1c-9b3d-2c1a7e5f0a91" \
  -d description="My first customer"

The rules, precisely:

It stores the outcome, not the request. Stripe saves the resulting status code and body of the first request for a given key — regardless of whether it succeeded or failed. A repeat returns the same result, including a 500. That surprises people, and it is the right call: if the first attempt genuinely errored, replaying the error is honest. Inventing a fresh attempt behind the same key would not be.

Client generates the key. A UUIDv4 or similar high-entropy string, up to 255 characters. If the server generated it, a retry would arrive with a new key and the whole mechanism would be pointless.

No sensitive data in keys. Not email addresses, not personal identifiers. Keys end up in logs, metrics and error reports.

Keys expire. Removed after at least 24 hours; reusing a pruned key produces a genuinely new request.

Parameters are compared. If the same key arrives with different parameters, that’s an error — it means the client made a mistake, not that it retried.

But the replay has a boundary. Anything rejected before the endpoint starts running is not covered. Rate limiting is the clearest case: Stripe’s own docs note that a 429 can produce a different result under the same key, because the rate limiter runs ahead of the idempotency layer. The same applies to a 401 with a missing key. Their advice for 4xx generally is blunt — generate a new key rather than assume the old result is authoritative.

And the subtle one: results are only saved after execution of the endpoint begins. If the request fails validation, or conflicts with another request executing concurrently, no idempotent result is stored, and the client may retry.

One small implementation detail worth copying: Stripe marks a replayed response with the header Idempotent-Replayed: true. It costs nothing and it turns “did my retry actually do anything?” from a debugging session into a header check.

That concurrency rule is where the real engineering is.

The Race That Breaks Most Implementations

Here is the naive implementation, and some version of it is running in production at a company near you:

# BROKEN — do not ship this
existing = db.get_idempotency_record(key)
if existing:
    return existing.response          # replay
result = charge_the_customer(params)  # ← two requests can both be here
db.save_idempotency_record(key, result)
return result

The window between the lookup and the save is small. It is not zero. Two retries arriving 40 milliseconds apart — entirely normal when a mobile client times out and immediately reconnects — both find nothing, both proceed, and the customer is charged twice.

The root error is thinking in two states. A key is not merely present or absent. It has three states:

That middle row is not an invention. Stripe’s own status-code reference lists 409 Conflict as “the request conflicts with another request (perhaps due to using the same idempotent key)”.

The fix is to make claiming the key an atomic operation, and to do it before doing any work:

-- Claim atomically. Either you inserted the row, or somebody else owns it.
INSERT INTO idempotency_keys (key, tenant_id, request_fingerprint, state, created_at)
VALUES ($1, $2, $3, 'in_flight', now())
ON CONFLICT (tenant_id, key) DO NOTHING
RETURNING id;

Zero rows returned means you did not win the race. Read the existing row: if it’s complete, replay the stored response; if it’s still in_flight, return 409 and let the client come back.

Then — and this is the part people skip — write the outcome in the same transaction as the business change:

BEGIN;
  INSERT INTO payments (...) VALUES (...);
  UPDATE idempotency_keys
     SET state = 'complete', status_code = 201, response_body = $1
   WHERE tenant_id = $2 AND key = $3;
COMMIT;

If the charge and the key record commit separately, you have simply moved the race rather than removed it. Crash between the two writes and the money moved while the key still says in_flight.

Two details that stop this from becoming its own outage:

Reap stale in-flight rows. A process that dies mid-request leaves a key stuck in in_flight forever, and that key will reject every retry until someone notices. Give in-flight records a lease with a timeout.

Store a fingerprint of the request. Hash the relevant parameters. If the same key arrives with a different fingerprint, reject it — the client has a bug, and silently replaying an unrelated response would be worse than an error.

Choosing the Key

Scope it to the caller. Always. A key alone is a global namespace, which means one tenant can collide with another’s key — and, worse, can probe for one. The unique constraint should be on (tenant_id, key), never key on its own.

Let the client generate it. Before the first attempt, reused unchanged across retries. That is the entire contract.

Do not hash the request body into a key. It is tempting: hash the payload and call it a key. But two legitimately identical requests — the same customer buying the same coffee twice in one minute — would collapse into one, and you would silently lose a real order. Content hashing detects duplicates; an idempotency key identifies one intended operation. Different jobs.

Deriving the key from a stable object the user is already acting on — a cart ID, an order draft ID — is a different matter and perfectly sound; Stripe suggests it explicitly. The cart identifies the operation. The payload merely describes it.

Do not put meaning in it. No account numbers, no emails, no timestamps you plan to parse later.

This Is Not Just an HTTP Problem

The same principle governs anything that can deliver twice, which is everything.

Message brokers are the obvious case: at-least-once delivery is the norm, so a consumer must be prepared for the same message twice. The mechanism is identical — a deduplication key, stored atomically alongside whatever the consumer changed. I went through this in detail when working out how to send a million notifications without falling over; the dedupe table there is an idempotency table wearing a different hat.

Which brings us to the phrase that causes the most confusion in distributed systems.

Exactly-once delivery does not exist. Over an unreliable network, the sender cannot distinguish “the request was lost” from “the response was lost”. It must choose: give up (at-most-once) or retry (at-least-once). There is no third option at the transport layer.

Note the word delivery. Systems that advertise exactly-once semantics — Kafka’s transactional producer being the famous one — are not breaking that rule. They achieve it by doing exactly what this article describes: sequence numbers, deduplication and atomic commits on the receiving side. The guarantee is real. It is built, not transported.

What you can build is effectively-once:

at-least-once delivery + idempotent processing = the effect happens once

That reframing is the practical payoff. Stop trying to make delivery perfect. Make repetition harmless.

What Good Looks Like

• Every state-changing POST and PATCH accepts an idempotency key; GET, PUT and DELETE don’t need one
• The key is claimed atomically before any work begins, not checked and then acted on
• The outcome and the business change commit in the same transaction
• Keys are scoped to the authenticated tenant, with a unique constraint on the pair
• in_flight records have a lease and get reaped
• A parameter fingerprint is stored, and mismatches are rejected rather than replayed
• Replays are labelled, so a client can tell a stored result from a fresh one
• Retention is documented, indexed, and actually enforced
• Someone has tested the concurrent case — two identical requests fired simultaneously, not sequentially

That last one is the difference between an implementation that works and one that appears to work. Sequential tests pass on broken code. Write the test that fires both requests at once.

The Bottom Line

Idempotency is not a header. The header is just how two systems agree on a name for an operation.

The actual idempotency lives in your storage layer: in a unique constraint, in an atomic claim, in a transaction boundary. If those are wrong, no amount of correct header handling will save you — and if they’re right, you are safe even from clients that retry far more aggressively than the spec permits.

RFC 9110 tells clients not to auto-retry non-idempotent methods. They will anyway. Build for the world you have.

The same instinct runs through choosing between a monolith and microservices: the failure modes you design for are the ones that don’t wake you up.