Delivery Semantics: At-Most, At-Least, Exactly-Once

A customer taps Place order once. Your payment service picks up the order.placed event, charges their card $42, and then — in the half-second before it can acknowledge the message — the pod dies. A deploy rolled out, or the box ran out of memory, or a spot instance got reclaimed. It doesn't matter which. What matters is that the broker never heard back.

So the broker does the one thing it promised to do: it assumes the message wasn't handled and redelivers it. A fresh payment worker picks up the very same event and charges the card again. One order, two $42 charges, one customer who is now writing an angry review. Nothing was buggy. Every component did exactly what it was designed to do. The double charge is a property of the design, not a mistake in it.

Here's the twist that makes this a genuinely hard problem. Suppose you try to fix it by acknowledging the message first, before charging. Now on the unlucky day the pod dies right after the ack and right before the charge, the customer is charged zero times — the order is silently lost, no error anywhere. You've traded a double charge for a vanished order. You cannot avoid both by choosing when to acknowledge, and that dilemma is the whole subject of this lesson.

In this lesson: the one decision (when to ack) that forces the choice between losing and duplicating, the three delivery guarantees that fall out of it, why "exactly-once" is a lie you'll still learn to deliver, how to publish an event without losing it, and what to do with a message that can never be processed.

Scope: the idempotency-key mechanics themselves live in their own lesson, Idempotency: Safe to Retry — here we use idempotency as the fix, not re-derive it. And change data capture gets only a taste; its full treatment comes later in the course.

Strip a reliable consumer down to its bones and it does three things in a row: it receives a message, it does the work (the side effect — charge the card, insert the row, send the email), and it acknowledges the message so the broker knows it's safe to stop redelivering. The entire subject of delivery semantics is one question about that sequence: do you ack before the work, or after? There is no third position, and each of the two has a failure window you cannot close.

Ack before the work. The moment you receive the message, you tell the broker "got it," and only then start the work. This is fast — nothing waits, nothing retries. But now imagine the crash lands between the ack and the work finishing. The broker has already been told the message is handled, so it will never send it again. The work never happened. The message is lost. You will never duplicate, but you can drop.

Ack after the work. You do the work first, and only acknowledge once it's done. Now no crash can lose a message — if you die before acking, the broker never got the ack and will redeliver. But that's exactly the problem: if you crash after the work but before the ack (or the ack itself gets lost on the network), the broker redelivers a message whose work already ran, and you do it again. You will never lose, but you can duplicate.

That's the trap in one line: ack early and you risk losing; ack late and you risk duplicating. There is no ordering of those three steps that gives you neither, because the crash can always fall in the one gap you left open. Everything else in this lesson is a response to this single, unavoidable fork.

Those two ack choices have names, and there's a mythical third everyone wants. Together they're the three delivery guarantees, and picking one is a real engineering decision with a real cost.

At-most-once — ack before the work (or just fire and forget). Each message is delivered zero or one times; it may be lost, never duplicated. It's the fastest and simplest — no waiting for acks, no retry logic, lowest latency. You choose it when losing a little is genuinely fine: a stream of CPU metrics, debug logs, a "user is typing…" indicator, a presence ping. Miss one and nothing breaks.

At-least-once — ack after the work. Each message is delivered one or more times; it's never lost, but it may be duplicated. This is the default for almost everything that matters — orders, payments, notifications, inventory updates — because losing them is unacceptable and duplicates are something you can handle. The price of "never lose" is "sometimes twice," and the rest of this lesson is about paying that price gracefully.

Exactly-once — each message takes effect once and only once, no loss, no duplicate. It's what everyone actually wants, especially for money: a payment, a ledger entry, a trade. It is also the most expensive and, as the next section explains, the most misunderstood — because in the strict sense it doesn't exist. You reach for it when a duplicate is a real second charge and the downstream can't protect itself.

A useful way to hold them: at-most-once optimizes for latency and tolerates loss; at-least-once optimizes for not losing and tolerates duplicates; exactly-once optimizes for correctness and pays for it in complexity. There's no universally right answer — only the right answer for this message.

The dilemma is easy to nod along to and genuinely surprising to watch. Here's the pipeline — a producer, a broker, a consumer that charges a card, and a ledger that counts the charges — with the failure entirely in your hands.

Try this. Set the consumer to ack after processing (at-least-once) and inject the crash after the charge but before the ack — then run it and watch the broker redeliver, the second consumer charge again, and the ledger land on 2. That's the double charge from the top of the lesson, made real. Now switch to ack before processing (at-most-once) and crash before the charge: the ledger lands on 0 — the order silently lost. Neither timing gave you exactly one. Finally, go back to at-least-once and flip on the idempotent consumer: the redelivered message is recognized by its id and skipped, and the ledger settles on 1. Same crash, same redelivery — one charge. That's the whole answer, and you just built it.

Time to be honest about the third promise. Exactly-once delivery is impossible in a distributed system, and not as a matter of engineering effort — as a matter of proof. It's the Two Generals Problem: over a network that can drop any message, two parties can never become certain the other received the last one. The sender that just did some work can't tell the difference between "my ack was lost" and "the receiver never got it," so it must either retry (risking a duplicate) or give up (risking a loss). No amount of extra messages closes that gap — each new confirmation is itself a message that can be lost. As the industry joke goes: the two hardest problems in distributed systems are exactly-once delivery, guaranteed ordering, and exactly-once delivery.

So how does anyone charge a card exactly once? By moving the guarantee to where it can live: processing, not delivery. The trick is to accept that the message may arrive more than once and make the second arrival do nothing. That's idempotency, and it sits squarely on the consumer: before charging, check whether you've already processed this event's id; if you have, skip it. The delivery is at-least-once; the effect is exactly-once. This combination even has a name — effectively-once — and it's what every system that claims "exactly-once" is actually doing under the hood.

This reframes the whole problem in a liberating way. You stop trying to prevent duplicates (impossible) and start making them harmless (very possible). Even Kafka's much-advertised "exactly-once semantics" fits here: it gives you a genuinely atomic read-process-write within Kafka — but its transactions cover only data inside Kafka. The instant your consumer writes to a database or calls a payment API, Kafka's guarantee stops at the boundary and idempotency is back to being your job. Exactly-once isn't a setting you turn on. It's at-least-once, plus a consumer that doesn't flinch when it sees the same message twice.

There's a quieter place the same problem hides, and it bites teams constantly: the moment a service has to both change its database and publish an event about it. Checkout writes the order row and publishes order.placed. Those are two different systems — your database and your broker — and you cannot wrap them in one transaction. So there's a gap, and the crash finds it:

• Write the database, then crash before publishing → the order exists but no event was ever sent. Inventory, email, and analytics never hear about it. A lost event.
• Publish the event, then crash before the database commit → everyone reacts to an order that doesn't exist. A phantom event.

This is the dual-write problem, and the tempting fix — a distributed transaction across the database and the broker — is a trap of its own: most brokers don't support it, and where it exists it's slow and couples the two systems tightly. Don't reach for it.

The clean solution turns two writes into one. With the transactional outbox, your service writes the order row and inserts the event into an outbox table in the same local database transaction — so by the database's own atomicity, either both land or neither does. A separate relay process then reads the outbox and publishes to the broker (at-least-once, naturally), marking each row as sent. The database and the event stream can no longer disagree. On the receiving end, the mirror image is the inbox: the consumer records each processed event id in a table in the same transaction as its side effect, so a duplicate is recognized and a mid-crash rolls back cleanly. Outbox to publish reliably, inbox to process once — the two halves of effectively-once, built entirely from at-least-once and idempotency.

There's an even slicker version worth knowing the shape of: your database already keeps a write-ahead log of every committed change, in order. Change data capture reads that log and turns each row change into an event — the outbox pattern without an outbox table, with the database's own log as the source of truth. It's a whole topic of its own, and you'll meet it properly later; for now just hold the idea that the database's log can be your event stream.

At-least-once means retries, and retries have an evil twin: the message that will never succeed no matter how many times you try. Malformed JSON, a field that violates the schema, a reference to a record that was deleted, an input that trips a specific bug — a poison message. Retry it forever and it does two ugly things: it burns resources looping, and it can clog the queue behind it, starving every healthy message waiting its turn. Your retry safety net has become a denial-of-service on yourself.

The fix is a dead-letter queue (DLQ): cap the number of attempts — usually somewhere around five to ten — and once a message blows past that, stop retrying it on the main queue and move it to a separate, quiet queue where failed messages are parked. Nothing is lost (it's safely in the DLQ) and nothing is blocked (it's out of the main line). Then a human, alerted by the DLQ's depth (watch that metric, like any queue), inspects the parked messages, fixes the bug or the bad data, and redrives them back to be reprocessed.

The judgment call underneath is distinguishing two kinds of failure. A transient failure — the payment API was briefly down — deserves a retry, ideally with backoff so it has room to recover before the attempt counter runs out. A poison failure — the message itself is broken — deserves the DLQ, because no number of retries can help. Backoff is what tells them apart in practice: with no delay between attempts, a genuinely poison message reaches the DLQ in milliseconds (fine), but a momentarily-overloaded downstream gets dead-lettered far too soon (not fine). Space your retries, cap them, and give the hopeless ones a quiet place to be examined.

The delivery-semantics myths are unusually sticky. Trade them in:

• "Exactly-once delivery is a feature I can turn on." → It isn't, and can't be — the Two Generals Problem forbids it over an unreliable network. What you can turn on is effectively-once: at-least-once delivery plus an idempotent consumer. Anyone selling you "exactly-once" is selling you that, whether they say so or not.
• "At-least-once is unsafe because it duplicates." → At-least-once is the safe default — it never loses. Duplicates are a known, handleable consequence you neutralize with idempotency. The unsafe choice is at-most-once used where loss actually matters.
• "Kafka's exactly-once makes my whole pipeline exactly-once." → Only the part inside Kafka. The moment your consumer charges a card or writes to a database, you're across the boundary and back to at-least-once + idempotency.
• "I'll just publish the event right after I save to the DB." → That's the dual-write problem waiting to happen — a crash between the two loses or phantoms the event. Use the outbox (one transaction), not two hopeful writes.
• "A dead-letter queue means I lost the message." → The opposite — the DLQ is how you keep a message that can't be processed yet, out of the way, until someone can fix and redrive it. Losing it would be silently dropping it.

• Delivery semantics come down to one decision: when do you ack? Ack before the work risks loss (at-most-once); ack after risks duplication (at-least-once). No timing avoids both.
• At-most-once (lose-tolerant, lowest latency) for metrics and logs; at-least-once (never lose, may duplicate) as the default for everything that matters; exactly-once for money — with the fine print below.
• Exactly-once delivery is impossible (Two Generals). What you build is effectively-once = at-least-once + an idempotent consumer — you stop preventing duplicates and start making them harmless. Kafka's exactly-once covers only data inside Kafka.
• Publishing an event reliably is the dual-write problem: don't do two hopeful writes. Use the transactional outbox (business change + event in one DB transaction) and an inbox / idempotent consumer to process each event once. The database's own log (change data capture) can even be the event stream.
• A poison message that can never succeed goes to a dead-letter queue after a capped number of retries — kept, not lost, and out of the way — while transient failures retry with backoff.

That closes out real-time and async communication: you can now move work off the request path (queues), choose how it's stored and consumed (broker vs log), broadcast it to many listeners (pub/sub), and reason honestly about how many times it actually happens (delivery semantics). Next is the section checkpoint — a few scenarios where you'll pick the messaging design and the delivery guarantee, and defend the trade-off.