Idempotency: Safe to Retry

The last lesson ended mid-nightmare, so let's live it properly. A customer taps Pay $49. The spinner spins. Thirty seconds later: request timed out. Now answer one question: was the card charged?

You can't. And neither can their app. From the client's side, a timeout is perfect silence, and silence is consistent with three completely different worlds: (1) the request never reached the server; (2) the server crashed mid-work; (3) the server did everything — charged the card, wrote the order, sent the receipt — and only the response got lost on the way back. Three worlds, one symptom, no way to tell them apart from outside. That's what makes a timeout ambiguous, and the ambiguity isn't exotic: responses die routinely — a mobile network hiccup, a load balancer draining the very box that served you (§5!), a deploy, one dropped packet on the return path.

So what should the app do? Not retrying risks world (1): no charge, no order, a lost sale and an angry customer. Retrying risks world (3): the charge already happened, and a second request means a double charge. Both options are wrong. Every engineer meets this fork eventually — usually via its most famous costume, the double-clicked Pay button, which is the same bug with zero network required.

Here's the reframe this whole lesson hangs on: the problem isn't the retry, and it isn't the timeout. You will never make the network reliable enough to remove the ambiguity. The move — the only move — is to make the ambiguity harmless: engineer the operation so that doing it twice has the effect of doing it once. That property has a name — idempotency — and once an operation has it, every retry stops being a gamble and becomes free reliability.

Lesson one planted this exact seed, so collect it now. HTTP's verbs come with formal retry guarantees:

• GET is safe — retry forever, nothing changes.
• PUT is idempotent by definition: PUT /users/42 {"email": "x@y.dev"} twice leaves the same final state as once. The second write overwrites the first with identical bytes.
• DELETE is idempotent: DELETE /reviews/9 twice = the review is gone either way (the second attempt just gets a 404 — the state didn't change twice).
• PATCH is the fine print: PATCH {"qty": 5} (set a value) is idempotent; PATCH {"op": "increment"} (apply a delta) is not. Same verb, opposite guarantees — hold that thought, it comes back at the end as a design principle.

Which leaves POST — "create a new one in the collection" — the verb whose entire meaning is non-idempotent. Two POSTs = two new things. And by cruel arrangement of the universe, POST is exactly where the irreversible stuff lives: charges, orders, signups, emails, webhooks. The verbs that are safe to retry are the ones that rarely need it; the verb that desperately needs a retry guarantee is the one HTTP can't give it to.

So we build the guarantee ourselves. The pattern that does it is one of the most elegant in API design, it fits in a single HTTP header, and you've already used it today if you bought anything online.

The trick is to give the server a way to recognize a repeat — and the insight is what gets named. Not the HTTP request. The intent.

Client side: when the app decides "this customer is buying this cart" — one logical operation — it mints a unique random key (a UUID) and attaches it to the request as a header:

POST /charges
Idempotency-Key: 91f4c8a2-77b1-4f0e-9e3a-2b8d1c5f6a90

{ "amount": 4900, "source": "card_x" }

The rule has two halves, both essential: every retry of this operation carries the same key (the intent didn't change — it's still that customer buying that cart), and every new operation mints a fresh key (a second, genuine purchase is a different intent). The key answers the one question the server couldn't answer before: "is this a repeat, or a new wish?"

Server side: the first time a key arrives, execute normally — charge the card — and store the outcome against the key: status code, response body, everything. If the same key ever arrives again: do not execute. Look up the stored outcome and replay it, byte for byte. The retry gets the receipt the original earned; the card is charged exactly once; the client can't even tell it got a replay (Stripe, whose API popularized this pattern, marks replays with an honest Idempotent-Replayed: true header and prunes stored keys after 24 hours; the pattern itself is now being standardized as the IETF Idempotency-Key header draft).

Walk the ambiguous timeout again with the key in place. Request sent, card charged, response lost — the client sees the timeout, shrugs, and retries with the same key. The server sees a key it has already completed, skips execution, replays the saved receipt. The user gets their confirmation; the ledger holds at one charge. The ambiguity didn't go away — it stopped mattering. That's the whole trick: same request twice, same result once.

One more guardrail worth copying from Stripe: if a key arrives with different parameters than its original (same key, different amount?), that's not a retry — it's a bug in the client. Reject it loudly. The key is scoped to one exact intent.

This is the rare lesson where you can commit the actual sin, watch the actual damage, then install the fix and watch it hold — so here's a checkout, a deliberately flaky network, and the two things worth staring at: the charges ledger (the ground truth) and the server's key store, both live.

Start with the key off. Click Pay $49, then use force response-loss so the next request does its work but the reply dies in the gap — you get the timeout, but watch the ledger: the charge landed. Now do what every human does: click Pay again. Two charges. No error anywhere, both requests handled "correctly," customer charged twice — you've just manufactured the exact support ticket that pays for this lesson.

Turn the key on and run the identical disaster. The retry carries the same key; watch the key store — the server finds it completed, refuses to execute, and replays the stored receipt (Idempotent-Replayed: true). Ledger: one charge, customer has their confirmation. Then try the double-click: two requests with the same key in flight simultaneously — watch one win the store's unique constraint and execute while the other bounces off with a 409 and picks up the replay a beat later. Still one charge. Finally click Pay for a new purchase and notice it mints a new key and charges normally — the key names the intent, not the button.

The client half is a header. The server half is where the real engineering lives, and it has one trap so common it deserves its own paragraph.

The obvious implementation reads: check if the key exists; if not, execute; then save the key. This is a race, and it's the same disease we dissected in the rate-limiting lesson — check-then-act. Picture the double-click: two identical requests arrive within milliseconds. Both check the store. Both find nothing. Both execute. Two charges — with idempotency keys, implemented sincerely, still broken. Any implementation where "look" and "claim" are separate steps loses this race under exactly the concurrency it was built to survive.

The honest fix is to make claiming the key atomic, and the database already knows how: a unique constraint on the key column.

INSERT INTO idempotency_keys (key, status)
VALUES ('91f4c8…', 'processing')
ON CONFLICT DO NOTHING;

Exactly one request wins that insert — atomicity by construction, no application-level cleverness required. The winner executes; the loser knows the key is taken and behaves accordingly. Around that atomic core, four more pieces complete a production-grade store:

• A status column — processing → completed. The concurrent loser finds the key processing: there's nothing to replay yet, so respond 409 Conflict ("in flight — try again shortly"); the client's next attempt finds completed and gets the replay.
• Store the outcome, not just the key — status code and serialized body, written when execution finishes. A key with no stored response can't replay anything.
• A TTL — Stripe prunes after 24 hours. Old keys cost storage forever otherwise; after pruning, a reused key executes fresh (acceptable: nobody retries a checkout from last month).
• A crash policy — if the winner dies mid-work, its key is stuck processing forever and blocks all retries. Real systems add a lock-steal: processing older than N minutes ⇒ the worker is presumed dead, a retry may take over. (What "take over" means when half a charge already happened is a genuinely deep problem — sagas and recovery live in a later container; today, know the hole exists.)
• Gold standard: insert the key and write the business row in the same database transaction — the claim and the work commit or vanish together.

Keys are the general-purpose fix, but the best idempotency is the kind you don't have to build — operations that are idempotent by shape. The dividing line is one you've now seen twice:

Say the desired state, not the change.

• SET balance = 90 — run it five times, balance is 90. Idempotent. DECREMENT balance BY 10 — run it five times, you've stolen $40. Not.
• status = "cancelled" — a fact, repeatable. cancel() that emails the customer and restocks inventory each call — a side-effect chain, not repeatable.
• PUT the whole resource (state) vs PATCH {"op": "increment"} (delta) — the L1 fine print, now revealed as a strategy.

When you control the contract, prefer state-shaped operations — they carry their own idempotency, no store, no TTL, no races. And there's one more elegant trick in this family: client-generated IDs. Instead of POST /orders (server picks the id, every call creates), design PUT /orders/{uuid} where the client mints the id: the first PUT creates, every retry overwrites the same row with the same bytes. The resource's own identity does the work — the URL is the idempotency key. Plenty of serious systems (and most sync protocols) lean on exactly this.

Keys, then, are for the operations that stay irreducibly effect-shaped no matter how you model them — charge the card, send the email, fire the webhook. Which, notice, is a shrinking set once you start designing with state-shaped verbs on purpose.

Idempotency is the precondition; the thing it unlocks is the retry, and retries have their own craft — met briefly at the load balancer (§5: budgets, backoff, jitter — retries can amplify an outage) and covered in full when we hit asynchronous communication in the next section. Today, only the one-slide version, because it's the contract between the two ideas that matters:

Retries without idempotency are an outage amplifier. A flaky hour plus aggressive clients plus non-idempotent POSTs = duplicated orders at scale, at the worst possible time.
Retries on top of idempotency are free reliability. The same flaky hour becomes invisible: every ambiguous timeout resolves itself on the next attempt, replays absorb the repeats, and the ledger stays true.

That ordering — make repetition harmless first, then retry boldly — is the professional sequence. Teams that add retry middleware before auditing idempotency get to discover this lesson from the incident channel.

Idempotency keys are close to a free lunch, but close isn't free — the honest bill:

• A stateful store on the hot path. Every keyed request now does a read-or-claim against the key table before doing its job — a write and (on Stripe's scale) a nontrivial storage system with its own TTL sweeper. For most teams it's one indexed table; it's still one more thing that can be slow or down. (Fail-open or fail-closed when the key store is unreachable? For payments: closed. You've made this exact call before, at the rate limiter.)
• The client must cooperate. The pattern only works if callers actually persist the key across retries — a client that mints a fresh key per attempt gets zero protection. This is contract-design work: document it, SDK it, make the right thing the default.
• Scope choices are judgment. How long is a retry still "the same intent" (24h is Stripe's answer, not a law)? Is the key per-endpoint or global? What happens on replay of a request whose response format has since changed (versioning, two lessons from now, waves hello)?
• It's exactly-once effect, not exactly-once delivery. The request may travel five times; the work happens once. True exactly-once delivery across systems is a famously harder (arguably impossible) promise — the difference becomes central in the messaging section.

Against all that: the alternative is double charges. This is the cheapest insurance in the catalog.

• "A timeout means the request failed." A timeout means you don't know. The work may be complete with only the answer lost — and treating "unknown" as "failed" is precisely how double charges are minted.
• "We'll just disable the button after one click." The network double-submits without any button: client retries, LB failovers, flaky mobile links. UI debouncing is polish; the ambiguity is structural and lives below the UI.
• "Idempotency keys are just request deduplication — hash the params." Two customers ordering identical carts are two intents (two keys, two orders); one customer retrying is one intent (one key, one order). A parameter hash literally cannot tell those apart. The key names intent, which only the client knows.
• "Check if the key exists, then execute." Check-then-act — the double-click executes twice with matching keys. Only an atomic claim (unique constraint, ON CONFLICT) actually closes the race. Same lesson the distributed rate limiter taught: shared counter, atomic or broken.
• "We don't handle money, so this doesn't apply." Every POST that sends an email, creates an account, fires a webhook, or enqueues a job has the same three-worlds ambiguity. Money doesn't create the bug; money just makes it show up on a bank statement.

Twenty minutes, any language, and you'll have built the real thing — including losing the race and then closing it:

1. Build the vulnerable server. One POST /pay endpoint that appends {amount: 49} to a charges list and returns a receipt. Add sleep(2) before the append (work takes time), and make the handler drop the response 50% of the time after doing the work (return nothing / kill the socket). That's your ambiguous network.
2. Be the frustrated user. curl -m 1 -X POST localhost:8000/pay — the 1-second timeout guarantees you never see a response. Run it three times, like anyone would. Now GET /charges: three charges. You've reproduced the double (triple) charge with two flags.
3. Add the key. Accept an Idempotency-Key header; keep a dict {key: response}; on a known key, return the stored response without appending. Retry with a fixed key (-H "Idempotency-Key: abc") through the same flaky endpoint until you get a response. Check the ledger: one charge, no matter how many attempts.
4. Now lose the race. Fire two requests with the same key simultaneously (curl ... & curl ... & wait). With a naive if key in store check and that sleep(2), both slip past the check: two charges, same key. Feel that — it's check-then-act failing exactly as advertised.
5. Close it. Move the claim into an atomic step — in SQLite, INSERT INTO keys(key, status) VALUES(?, 'processing') with a UNIQUE column and catch the violation (return 409). Re-run the race: one executes, one bounces, ledger holds at one.

Step 4 is the one to remember. Everyone believes in the race after they've lost it on their own laptop.

The network hands you an unavoidable ambiguity, and idempotency is the discipline of making it not matter:

• A timeout is three worlds in one symptom — never-arrived, died-midway, or worked-but-the-answer-drowned. You cannot distinguish them from the client. Stop trying; make repetition harmless instead.
• HTTP gives you GET/PUT/DELETE retry-safe by spec; POST is the hole — and it's where charges, orders, and signups live.
• Idempotency keys close the hole: the client names the intent (same key on retry, new key for new intent); the server executes once, stores the outcome, and replays it for every repeat. Stripe canon: 24h TTL, param-mismatch rejection, Idempotent-Replayed: true.
• The server's claim must be atomic — check-then-act loses the double-click race; the unique constraint (ON CONFLICT DO NOTHING) plus a processing/completed status and a stored response is the pattern. Gold standard: key and business write in one transaction.
• Better still, design it away: say the state, not the change — SET over INCREMENT, PUT over effectful PATCH, client-generated IDs where the URL itself becomes the key.
• Then, and only then, retry boldly — idempotency first, retries second; the reverse order is an outage amplifier.

Notice what this section is quietly building: L1 made the contract guessable, L2 made walking it honest, and now L3 made repeating it safe. The next lesson turns to what's inside the messages themselves — JSON you can read but pay for in bytes, versus binary formats that are fast, tiny, and schema-checked: data formats and serialization, where Protobuf enters the story.