Outbox + CDC: Reliable Events from Your Database

This whole section started with a broken promise. In The Distributed Transaction Problem you saw the dual write: to both change your data and tell the outside world, you update your database and publish an event to a queue — two independent systems, with no shared transaction. A crash in the gap between them and you get an order with no event (the warehouse never ships) or an event with no order (a phantom). And Saga: The Data-Consistency Mechanics made it worse: choreographed sagas depend on those events arriving — a dropped "flight reserved" silently strands the whole workflow.

So how do you emit an event from a database reliably — never losing it, no matter when the process dies? The answer is almost annoyingly simple, and Chris Richardson (microservices.io) states it as a rule, not an optimization:

Write the business state and the event in the same database transaction, then publish it asynchronously from a separate component.

That's the whole idea. Stop writing to two systems. Instead of "commit the order, then publish the event" — two writes that can tear apart — you make the event part of the same commit as the data, by writing it into an ordinary table. Then a separate process ships it onward. One atomic write replaces the racing pair; the tear-in-the-middle is gone because there is no middle. The rest of this lesson is what that table is (the outbox), who ships the events (the relay — by polling, or by tailing the log with CDC), and the one catch you can't design away (at-least-once delivery). It all fits in a single picture:

Add one table. Call it outbox. It holds the events you intend to publish — each row is one event, with a unique event id, a payload, and a published flag:

CREATE TABLE outbox (
  id         bigserial PRIMARY KEY,
  event_id   uuid   DEFAULT gen_random_uuid(),   -- the dedup key (matters in §4)
  aggregate  text,
  payload    jsonb,
  published  boolean DEFAULT false
);

Now the move that makes it all work. In the one local transaction that changes your business rows, you also insert the event into the outbox — two inserts, one commit:

BEGIN;
  INSERT INTO orders  VALUES (1, 'laptop', 1200);
  INSERT INTO outbox (payload)
       VALUES ('{"type":"OrderPlaced","orderId":1,"amount":1200}');
COMMIT;

Because both inserts live in the same transaction, they are atomic together — the guarantee from ACID, Precisely. Either the order and its event both commit, or neither does. There is no longer any window where the data exists without the event. On real Postgres, the outcome is exactly that both-or-neither:

after COMMIT ................  orders = 1   outbox = 1
after a crash (ROLLBACK) ...  orders = 1   outbox = 1     ← the rolled-back attempt left NEITHER

The rolled-back transaction — an app that died before COMMIT — added nothing: no order, no event. The dual write's fatal gap is closed, and it's closed by the database's own atomicity, not by hope. Notice what just happened to the event: it rode into the database on the same commit — the same write-ahead-log record (WAL & Durability: Why Committed Data Survives) — as the data it describes. It is now durably recorded. The only job left is to get it from the outbox to the queue, and that job can fail, retry, and restart as many times as it likes without ever losing the event, because the event is safe in the table. That shipping job is the relay.

The relay is a separate process whose only job is: read committed outbox rows and publish them to the queue. There are two ways to build it, and the choice is a real tradeoff.

1 · Poll the table. The simplest relay is a loop: SELECT the unpublished rows, publish them, mark them done. To run several relay workers without any two grabbing the same row, use the pessimistic claim from Pessimistic vs Optimistic Locking:

SELECT id, payload FROM outbox
 WHERE NOT published
 ORDER BY id
 FOR UPDATE SKIP LOCKED   -- each worker claims a DIFFERENT batch
 LIMIT 100;
-- …publish to the queue…
UPDATE outbox SET published = true WHERE id = ANY(:ids);

Polling works anywhere — any database, no special features. The costs: it adds query load on your database, and it has polling latency (an event waits until the next poll). Tune the interval and you're trading freshness against load.

2 · Tail the log (Change Data Capture). Here's the elegant one, and it's the payoff of a spiral we planted long ago. Your database already writes every committed change to its write-ahead log — that's what makes commits durable (WAL & Durability). CDC simply reads that log and streams the changes to the queue. No polling, no extra queries — it consumes the log the database was writing anyway. The canonical tool is Debezium, which tails the PostgreSQL WAL (via logical decoding) or the MySQL binlog.

This isn't hand-waving — Postgres exposes the exact mechanism, so we can watch CDC work. Create a logical replication slot (what Debezium attaches to), make two committed changes and one that rolls back, then read the log:

SELECT pg_create_logical_replication_slot('cdc_slot', 'test_decoding');
-- INSERT id=5 (commit) · INSERT id=6 (ROLLBACK) · UPDATE id=5 (commit)
SELECT data FROM pg_logical_slot_get_changes('cdc_slot', NULL, NULL);

BEGIN
table public.orders: INSERT: id[integer]:5 item[text]:'mouse' amount[integer]:25
COMMIT
BEGIN
table public.orders: UPDATE: id[integer]:5 item[text]:'mouse' amount[integer]:30
COMMIT

Read that output closely, because it shows three things at once. The committed INSERT id=5 and UPDATE (amount 25 → 30) stream straight out of the WAL. They arrive in commit order. And the rolled-back id=6 is nowhere — CDC surfaces only committed changes, with zero polling of the table. Debezium is doing precisely this, continuously, and tracking an offset (how far through the log it has read) so it can resume after a restart. Log-based CDC has minimal impact on the source, captures every change type (insert/update/delete), and preserves exact order — which is why it wins at scale.

One subtlety: CDC can tail your outbox table, or it can tail your business tables directly (no outbox at all). The direct approach is tempting — one less table — but then your event schema is welded to your table schema: rename a column and you break every consumer. The outbox keeps an explicit, stable event contract, decoupled from how you happen to store rows. That decoupling is usually worth the extra table.

The outbox guarantees your event is never lost. It does not guarantee it arrives exactly once — and that gap is not a bug you can fix, so you'd better design for it.

Look again at the relay's two steps: publish the event, then mark the row published. Those touch two systems (the queue and the database) — which is the dual write all over again, just moved downstream. If the relay publishes, then crashes before it marks the row done, the row still looks unpublished. On restart, the relay finds it and publishes it again. The event goes out twice. So the real guarantee is at-least-once: never zero, sometimes more than one.

Can't we just make the relay publish-and-mark atomically and get exactly-once? No — and this is fundamental, not a tooling gap. Exactly-once delivery over an unreliable network is impossible — it's the two-generals result from The Distributed Transaction Problem: no finite exchange of messages can guarantee both sides agree a message arrived exactly once. (Kafka's "exactly-once semantics" is a closed-system trick inside Kafka; it does not extend to an arbitrary consumer reading from it.)

So we stop fighting delivery and fix processing instead. Make the consumer idempotent: processing the same event twice has no additional effect. The mechanism is the event id from §2. Every retry reuses the same id; the consumer keeps a processed_events table (event id as primary key) and refuses to act on an id it has already seen:

INSERT INTO processed_events (event_id) VALUES ('…') ON CONFLICT DO NOTHING;

The first time an event arrives, that insert takes; the second time — the duplicate — it's a no-op, and the consumer skips it. On real Postgres, delivering the same event twice reads exactly like this:

first delivery  ...  INSERT 0 1     ← new id, recorded → process it
duplicate       ...  INSERT 0 0     ← already seen → skip (a no-op)

INSERT 0 1, then INSERT 0 0. Same event, delivered twice, processed once. That is the whole trick, and it has a name: at-least-once delivery + an idempotent consumer = "effectively-once." You don't get exactly-once on the wire; you get exactly-once effects, which is the thing you actually cared about. (Idempotency is important enough that it's the entire next lesson — here it's just the consumer's half of the outbox contract.)

Three ideas, one moving system: an atomic write, a relay, an idempotent consumer. The way to feel how they fit is to run the pipeline and break it on purpose — because the whole value of the outbox only shows up at the moment of a crash.

Below is the live pipeline: App → Database → Relay → Queue → Consumer. First flip it to naive dual-write and crash the app right after the database commit — the event is simply gone, exactly the failure that opened this section. Now flip to the outbox and inject the same crash at the same spot: the event was committed with the order, so it's safe in the outbox, and the relay delivers it on recovery. Same crash, opposite outcome — that contrast is the lesson. Finally, crash the relay mid-publish and watch it re-publish on restart: the queue holds the event twice, and the idempotent consumer deduplicates it away. Hammer the tally and watch the naive path leak events while the outbox never does.

The decisions almost make themselves once the pieces are clear:

• Reliable events from a database → use the outbox. Always. It's not exotic microservices esoterica; it's the standard, boring, correct way to emit an event whenever you must change data and tell the world. The cost is one extra row per event (in the same transaction — cheap) plus running a relay.
• Relay by polling vs CDC. Start with polling when you want simplicity, have no special database features, and can tolerate a little latency — it works everywhere. Move to CDC (Debezium) when you need low latency, low database load, and exact ordering at scale, and you're willing to operate the connector and its log slot.
• Outbox table vs log-based CDC on business tables. Prefer the outbox table when you want a clean, stable event contract decoupled from your schema (almost always). Tail business tables directly only when you truly can't add an outbox and accept that your events mirror your table layout.
• Every consumer must be idempotent. At-least-once is the guarantee you get; dedup by event id is the price of admission. Never deploy a consumer that breaks when it sees the same event twice.

Step back and the shape is clean. The dual write failed because it wrote to two systems with no shared transaction. The outbox turns it into one atomic write (data + event, both-or-neither) plus a reliable relay (poll the table, or tail the WAL with CDC) — and closes the one remaining gap, duplicate delivery, not by chasing the impossible exactly-once, but by making the consumer not care. Never lost, maybe twice, processed once.

That last clause — processed once — has been doing a lot of quiet work. "Make the consumer idempotent" is easy to say and surprisingly full of traps: what's a good idempotency key, how long do you remember it, which operations are naturally idempotent and which you have to force. That's not a footnote; it's a discipline, and it's next — Idempotency as a Discipline.