The Distributed Transaction Problem

Pessimistic vs Optimistic Locking ended with a warning: every tool in it — SELECT … FOR UPDATE, version columns, deadlock detection — lives inside a single database. The moment the two things you need to change atomically live on different machines, all of it stops working. This lesson is about that wall.

Here's the whole problem in one sentence: transfer $100 from Alice's bank to Bob's bank. On a single database this is the easiest thing in the world — it's the BEGIN; debit; credit; COMMIT; of ACID, Precisely, and atomicity guarantees all-or-nothing for free. One write-ahead log, one commit, one undo. But Alice and Bob bank at different banks, and each bank runs its own database, with its own commit and its own failures. There is no shared BEGIN. There is no single COMMIT. The one logical change — move the money — now has to happen as two separate commits on two independent machines, and that is where the ground opens up.

(We'll keep saying "two banks" because it's the cleanest version, but this is the shape of an enormous class of real systems: any time one business operation must change data owned by two independent parties — two databases, a database and a payment provider, a database and a search index — you are here. In modern architectures the "two banks" are usually two services with two databases; the problem is identical, and we'll meet that framing later in the course.)

The naive plan is obvious: debit Alice at Bank A, then credit Bob at Bank B. Let's actually run it — on two independent PostgreSQL instances (Bank A holds alice = 100, Bank B holds bob = 0; total money in the system: $100). Step one, Bank A does its own transaction and commits:

-- Bank A
UPDATE accounts SET balance = balance - 100 WHERE name = 'alice';   -- alice = 0, COMMITTED

That commit is durable — it survives anything now (ACID, Precisely again). Step two, we go to credit Bob at Bank B — but at that exact moment Bank B freezes (a crash, or the network drops between the two banks — from Bank A's side, the two are indistinguishable). Here is the real, verbatim result:

psql: error: connection to server at "bankb" (172.17.0.3), port 5432 failed: timeout expired

Bank A can only wait, and then time out. Now look at the world: **Bank A alice = 0, Bank B bob = 0 → total = $0.** We started with$100 and now have nothing. $100 has vanished — and here's the cruelty: Bank A already committed the debit. It's durable. There is no undo. You can't roll back a transaction that already returned COMMIT. The money didn't move; it evaporated, and no amount of cleverness at Bank A can bring it back.

That gap between the two commits — where one has happened and the other hasn't, and a failure can strike — is the failure window, and it is the beating heart of the distributed transaction problem. It cannot be closed by trying harder; as we'll see, it can only be managed. But first, let's kill the fixes you're already reaching for.

Three fixes leap to mind. All three fail, and understanding why is half the lesson.

"Just wrap both in one transaction." There is no such transaction. A database transaction is bound to one connection to one database — we can prove it: open a transaction, then reconnect to another database and the original BEGIN is simply gone (You are now connected to database… and your uncommitted work vanished). Postgres has no syntax to BEGIN at Bank A and COMMIT at Bank B. The single atomic commit you're imagining does not exist across independent databases.

"Just retry until both succeed." Retrying is not just useless here — it's dangerous. If Bank A committed and Bank B failed, you can't retry the whole transfer, because that would debit Alice a second time — and you can't un-commit the first debit. And retrying just the credit runs into a worse trap we'll see in a moment: if Bank B did apply the credit but its reply was lost, your retry credits Bob twice. Retry without care turns "money lost" into "money duplicated." (The only thing that makes retry safe is idempotency — designing each step so applying it twice is the same as applying it once — a discipline we'll build later.)

"Just lock both rows." Locks are local (Pessimistic vs Optimistic Locking): Bank A's lock manager knows nothing about Bank B's. There is no shared lock manager across independent machines. And the moment you try to build one — hold a lock at both banks until both are ready — you've invented a coordination protocol whose failure modes are worse than the problem: if the thing coordinating the locks dies at the wrong instant, both banks sit holding locks forever, waiting for a decision that never comes. (That's a real protocol, and its blocking flaw is the whole of the next lesson.)

Every naive fix fails, and they fail for the same underlying reason — which is worth stating plainly.

All-or-nothing across independent, failing nodes means every participant has to agree on the same outcome — everyone commits, or everyone aborts. And agreement over an unreliable network is provably imperfect. This isn't an engineering shortfall; it's a law.

The clean intuition is the Two Generals' Problem. Two allied generals must attack at the same time to win, but they can only coordinate by sending messengers across a valley the enemy patrols — messengers who might not make it. General A sends "attack at dawn." Did it arrive? A needs an acknowledgement. But B's acknowledgement might be lost too — so B needs an acknowledgement of the acknowledgement. It never ends: no finite exchange of messages can make both sides certain. The killer version, for us: after Bank A sends "credit Bob," and hears nothing back, it cannot tell the difference between "Bob was never credited" and "Bob was credited but the reply got lost." You saw that exact ambiguity in the timeout expired above — the request may have arrived and done its work; the silence tells you nothing. You can only ever reach probabilistic agreement, never certainty.

Theory nails the coffin shut twice more. The FLP impossibility result (Fischer, Lynch, and Paterson, 1985) proves that in a fully asynchronous network, no deterministic protocol can guarantee agreement if even a single node might crash — and "commit or abort?" is an agreement. FLP is in a sense stronger than the CAP theorem from CAP, Properly: CAP says you're in trouble during a network partition, but FLP says the mere possibility that one node crashes is enough to prevent guaranteed agreement — no partition required. And CAP itself frames the practical choice: to commit atomically, every participant must be reachable and must agree; when one isn't, you must pick — block and wait (sacrificing availability) or proceed without them (sacrificing atomicity). There is no third door.

So the distributed transaction problem is not a bug to be fixed. It's a fundamental limit: you cannot get perfect, atomic, all-or-nothing agreement across independent machines that fail and communicate over an unreliable network. Everything the rest of this section teaches is a way of choosing which imperfection to accept.

You might think "two banks" is exotic — most of us aren't wiring money between institutions. But the identical problem shows up in the most ordinary system you'll ever build, the moment you do this: write to your database, then publish an event to a queue.

Save the order to Postgres, then publish an OrderPlaced message to Kafka so the warehouse ships and the customer gets an email. Two independent systems, two separate writes — a dual write. We ran it: saved the order (the app DB now has orders = 1), then the message queue froze, and the publish returned the same timeout expired. The result:

app DB orders: 1
queue events:  0

An order with no event. The customer was charged; the warehouse never hears about it; the email never sends. (Flip the failure and you get the mirror-image horror: the event fires but the DB write rolled back — now the warehouse ships an order that doesn't exist.) It's the two-banks transfer wearing a hoodie, and it is lurking in every event-driven system on earth. People hit it constantly without recognizing it as the distributed transaction problem — which is exactly why naming it matters. (The clean fix — write your data and your outgoing event in one local transaction, then relay the event separately — is a pattern we'll build in a later lesson.)

You now know the shape of the problem; the fastest way to believe it is to try every naive plan and watch each one fail. Below is the two-banks transfer as a live lab. Pick a plan — debit A first, credit B first, or debit-then-retry — and inject a failure: the second bank crashes before its write, or it writes but its reply is lost, or nothing fails at all. Run it and watch the money.

You'll find each plan has a way to break. Debit-first + a crash makes **$100 vanish.** Credit-first + a crash **creates**$100 out of nothing. Retry + a lost reply credits Bob twice — money duplicated. And the lost-reply case always leaves the app in doubt: the money might be fine, but you cannot tell, which is its own kind of disaster (do nothing and hope, or retry and risk doubling). The 3×3 matrix fills in as you explore — and the point lands on its own: the only cell that's ever safe is "no failure," and you cannot guarantee no failure. There is no safe naive plan.

If the problem can't be solved, what do you actually do? You have exactly three moves, and the rest of this section is a tour of them.

1. Coordinate. Appoint a coordinator that asks every participant to prepare ("can you commit? promise me you will") and, only once everyone has promised, tells them all to commit. This buys you real atomicity — all-or-nothing — which is why it's the classic answer. The price is that it blocks: while everyone is waiting on the coordinator's decision, they hold their locks, and if the coordinator picks the wrong moment to die, they wait forever. That's two-phase commit, and its promise and its blocking flaw are the very next lesson, Two-Phase Commit (and Why 3PC Exists).
2. Compensate. Give up on atomicity and instead do each step for real, one at a time — and if a later step fails, run an action that undoes the earlier ones (refund the payment, restock the item). This keeps the system available (nobody blocks), at the cost of temporary inconsistency (for a moment, the money was debited before the refund) and a lot of careful bookkeeping. That's the Saga pattern — Saga: The Data-Consistency Mechanics.
3. Avoid it. The move nobody teaches, and often the best one: arrange things so the change isn't distributed at all. If you can pull the two writes into a single owner with a single database, you're back to one boring BEGIN … COMMIT and the whole problem evaporates. A great deal of senior design is collapsing boundaries so that the operations that must be atomic together live together. The strongest engineers reach for a distributed transaction last, not first.

The takeaways:

• One logical change across two independent stores cannot be made perfectly atomic. There is no transaction, no commit, and no lock that spans two independent databases (proven: reconnecting ends a transaction; measured: a naive transfer took $100 to$0 with no undo).
• The hazard is partial failure, and its cruelest face is ambiguity: a timeout expired can't tell you whether the far side succeeded (the Two Generals problem). Naive fixes make it worse — retry double-applies, locks are local.
• It's a fundamental limit, not a bug. Atomic commit is an agreement problem, and agreement over an unreliable network is provably imperfect (FLP; and CAP forces block-or-proceed under partition).
• It's everywhere — the dual write (database + queue → an order with no event) is the same problem in every event-driven system.
• You get three moves: coordinate (2PC — atomic, but blocks), compensate (Saga — available, but temporarily inconsistent), or avoid it (one owner, one transaction — often the best). Next — Two-Phase Commit (and Why 3PC Exists): we take the first move seriously — build a coordinator that makes everyone agree, and then watch the one failure that brings the whole thing to its knees.