Idempotency as a Discipline

Everything in this section has been paying for the same thing. Two-Phase Commit bought atomicity by blocking; Saga: The Data-Consistency Mechanics and Outbox + CDC: Reliable Events from Your Database bought availability by accepting at-least-once delivery — a message can arrive more than once, and you can't stop it (exactly-once delivery is impossible — the two-generals result from The Distributed Transaction Problem). Every one of those designs left an IOU that comes due here: the thing that receives the message must not break when it sees it twice. That property has a name, and building for it is a discipline.

An operation is idempotent if performing it any number of times leaves the system in the same state as performing it exactly once. Formally, for the state change f: f(f(x)) = f(x). The word is from algebra — abs(abs(x)) = abs(x), max(a, max(a, b)) = max(a, b) — and in systems it is the property that makes repetition harmless. That matters because in a distributed system repetition isn't an edge case, it's guaranteed: at-least-once delivery redelivers, retries on an ambiguous timeout re-send, crashed consumers reprocess, cron jobs re-run, event logs get replayed. You cannot make the network deliver exactly once — so you make processing tolerate duplicates.

You met the first taste of this in Idempotency: Safe to Retry: an HTTP POST /charges times out, the caller can't tell "it worked, the reply was lost" from "it never ran," so it retries — and an idempotency key stops the double charge. That was one operation (an HTTP request) and one mechanism (a key store). This lesson promotes idempotency from a trick you reach for to a discipline you apply to every effect in the system — with a way to decide, for any operation, whether it's already safe, and if not, exactly what to add. It starts with a question you should ask before writing any code.

Before you build any dedup machinery, ask the cheapest question: is this operation already idempotent? Operations split cleanly, and the split decides everything.

Naturally idempotent — operations that assert an absolute STATE. SET balance = 100; UPDATE … SET status = 'shipped'; PUT /users/42 with the whole resource; DELETE /x (gone stays gone). Re-applying overwrites with the same value, so the second, third, and tenth attempt change nothing. On real Postgres, running balance = 100 twice leaves it at 100 — the shape carries the guarantee, no mechanism required.

Not idempotent — three families, each a repeat hazard. Delta / relative ops accumulate: balance = balance + 10 applied twice gives 120, a silent double-credit. Create ops duplicate: a second INSERT (or POST /orders) makes a second row. Side-effects that escape the database — send an email, fire a webhook, charge a card at a third party — repeat the real-world action, and worse, you can't roll them back.

delta:  balance += 10   →  applied twice  →  120   ✗ not idempotent
state:  balance = 100  →  applied twice  →  100   ✓ naturally idempotent

Here's the lever most people skip: when you control the contract, you can often convert a non-idempotent operation into an idempotent one — say the desired state, not the change. Prefer SET balance = 90 over -= 10; prefer letting the client supply the id (PUT /orders/{client-uuid} instead of POST /orders) so the resource id is the idempotency key and a retry is a harmless upsert. This is the best idempotency you can get, because it costs nothing to run — you designed away the need to remember anything. Reach for a mechanism only when the effect is irreducibly a create or a side-effect (a charge, an email, an order whose id you don't control).

When an operation is irreducibly non-idempotent, you make it safe one of two ways: remember what you've already done, or guard the write so a duplicate matches nothing.

Remember — a dedup table keyed by an idempotency key. Give every logical operation a stable key (a UUID the producer generates once and reuses on every retry). Keep a processed(event_id PRIMARY KEY) table, and claim the key atomically with the database's unique constraint:

INSERT INTO processed(event_id) VALUES ('evt-1') ON CONFLICT DO NOTHING;
--  INSERT 0 1  → first time: claimed, do the work
--  INSERT 0 0  → already there: a duplicate, skip it

That unique constraint is the only correct check. The tempting SELECT-then-INSERT is check-then-act — a race where two concurrent duplicates both find the key missing and both execute (the same disease as the double-click and the rate-limiter race). The atomic insert lets exactly one claimant win.

⭐ The gold standard: claim the key in the same transaction as the effect. If the claim and the business write are separate steps, you've rebuilt the dual write inside your consumer. Watch what the split costs, on real Postgres — apply first, record the key second, and crash between:

deliver evt-2 → UPDATE balance += 10 → 110
💥 crash before INSERT processed('evt-2')
redeliver evt-2 → key not found → applies AGAIN → 120     ✗ duplicate

(Record the key first and crash before the effect, and you get the opposite bug — lost work, with the dedup now suppressing the retry that would have fixed it.) Both splits are wrong; one transaction is the fix — the atomicity of ACID, Precisely. Done right, the claim and the effect are one commit, and a redelivery is absorbed: evt-1 applies once to 110, and the duplicate is a no-op — UPDATE 0, still 110.

Guard — fold "have I done this?" into the write itself. Sometimes you don't need a dedup store at all; you phrase the effect as a conditional write so a duplicate matches zero rows. Two shapes, both from earlier lessons:

• Compare-and-set on a version (the optimistic move from Pessimistic vs Optimistic Locking): UPDATE … SET balance = balance + 10, version = version + 1 WHERE id = 1 AND version = $expected. The first delivery matches and bumps the version → UPDATE 1; a duplicate carries a stale version and matches nothing → UPDATE 0. The version is a fencing token — it also rejects a delayed duplicate from an old attempt.
• A status guard (the saga semantic lock from Saga: The Data-Consistency Mechanics): UPDATE orders SET status = 'CANCELLED' WHERE id = 1 AND status = 'PENDING' → UPDATE 1 then UPDATE 0. The compensation is idempotent for free.

Guards are cheaper than a dedup table (no extra store, naturally atomic) but only work when the row already exists and the effect is a guarded state transition. Creates and external side-effects still need a key — or a key pushed downstream, which is the first of the hard parts.

"Just add a dedup table" is where most designs stop and most bugs begin. The discipline is in four details that are easy to get wrong.

The key identifies the INTENT — not the attempt, not the payload. The key names one logical operation, so every retry of it reuses the same key and two genuinely different operations get different keys — even when their bodies are identical. A customer legitimately buying the same $9 item twice is two keys and two charges; the same purchase retried is one key and one charge. Hashing the request body can't tell these apart — it would merge the two real purchases or fail to catch a retry whose body shifted by a byte. The key is semantic, minted once at the source of intent (the click, the producer's event id) and carried unchanged through every hop.

⭐ You can't remember forever — so mind the retention WINDOW. A dedup table grows without bound, so you prune old keys on a TTL. But then dedup is only correct inside the window, and the rule is unforgiving:

the retention window must be ≥ the maximum possible redelivery delay of a duplicate.

That maximum is the sum of the client's retry budget, the broker's redelivery timeout, a paused or lagging consumer, and any manual replay of the log. Prune a key before its last straggler can arrive and that late duplicate looks brand-new and re-executes. We can watch it happen: a duplicate that arrives while the key is still remembered is absorbed (UPDATE 0, balance stays 110); prune the key, and the very next redelivery of the same event slips through — UPDATE 1, balance jumps to 120. Pick the window from the slowest duplicate you must survive, not the average. (Guards sidestep this entirely — a version has no TTL to misjudge.)

Un-rollbackable side effects live outside your transaction. All of the above borrows its atomicity from the database. But an email, a webhook, a charge at Stripe is a second system — you can't put it inside your COMMIT, and you can't un-send it. The move is to push idempotency downstream: pass your stable key as the provider's idempotency key, so even if you call twice, they act once. There is no way to make "send exactly one email" atomic with a local write; the honest design is at-least-once send plus a downstream idempotency key equal to one email effect.

Idempotency is not ordering. Repeat-safe is not the same as reorder-safe. If two different updates to the same entity arrive out of order (at-least-once brokers don't promise order), dedup won't save you — that needs a version or a sequence, not a dedup table. Conflating "I deduped, so I'm consistent" is a classic mistake: dedup fixes duplication, not reordering.

The whole discipline is one claim: whether a duplicate hurts depends on the operation's nature AND the mechanism you add. That's much easier to feel than to read, so build the consumer and attack it. Below, a message arrives at a consumer over an at-least-once channel. Pick what the message does and how the consumer protects itself, then deliver a fresh event and redeliver a duplicate.

Run the obvious combinations. A delta with no mechanism: the duplicate double-charges to $120 — corrupted. Add a **dedup table** and the same duplicate is absorbed at$110. Now switch the operation to a state write and drop the mechanism entirely: the duplicate lands right back on $90 — naturally idempotent, no machinery needed (this is the ask-first lesson, made concrete). Then the sharp one: with the dedup table working, hit prune + straggler and watch a late duplicate slip through the moment the key is forgotten. Same duplicate, wildly different outcomes — the discipline is choosing correctly on both axes.

Step back and the whole section snaps into one line. You cannot buy exactly-once delivery — the network won't sell it (two generals). So you stop trying to make the wire perfect and make the effect perfect instead:

effectively-once = at-least-once delivery + idempotent processing.

Deliver a message as many times as the network forces, and process it so only the first delivery changes anything. Idempotency isn't a nice-to-have bolted on at the end — it is the precondition that makes every at-least-once system in this section correct. A retry on a non-idempotent operation is an outage amplifier; the same retry on an idempotent one is free reliability. That's why Outbox + CDC insisted every consumer be idempotent, why a Saga's steps and compensations must be retriable, and why the safe answer to the ambiguous timeout was never "retry" alone.

The discipline, as a checklist — for any operation that can be retried or redelivered:

1. Is it already idempotent? A state/absolute write (SET, PUT, DELETE) is — ship it. A delta, create, or side-effect is not.
2. Can you convert it? Say the state instead of the change; let the client supply the id. The cheapest idempotency is design.
3. If not, remember or guard. A dedup table keyed by the intent (ON CONFLICT DO NOTHING), or a conditional write / version guard (WHERE version = $v). Claim the key in the same transaction as the effect.
4. Mind the window. Retention ≥ the slowest possible duplicate, or a straggler slips through.
5. Handle the boundary. For un-rollbackable side effects, push the idempotency key downstream.
6. Remember what it isn't. Idempotency fixes duplication, not ordering.

This closes the mechanics of transactions and concurrency. You've built up every tool: atomicity and isolation, locking optimistic and pessimistic, the distributed-transaction wall and its two answers (coordinate vs compensate), reliable events, and now the discipline that makes retries safe. Next, it all gets put in one fire at once — Flash Sale: Inventory Under Contention, where thousands of buyers hit the last hundred units in the same second, and every idea in this section either holds or breaks under the load.