Saga: The Data-Consistency Mechanics

The Distributed Transaction Problem gave you three moves; Two-Phase Commit took the first one — coordinate — and found its fatal flaw: to stay atomic, it holds everyone's locks and blocks when the coordinator dies. This lesson is the opposite bet. Instead of holding one big lock across everyone until you're sure, a saga does the unthinkable: it lets each step commit for real, as it goes, holding no cross-service lock at all — and keeps a plan to undo the earlier steps if a later one fails.

The idea is old and precise. Hector Garcia-Molina and Kenneth Salem named it in a 1987 paper, Sagas, to solve a different pain — a long-lived transaction that holds locks for minutes or hours starves everyone else. Their fix: don't run it as one giant locked transaction; run it as a sequence of small local transactions, each of which commits and releases immediately, plus a compensating transaction for each step that can semantically undo it. The guarantee a saga gives you is subtly different from ACID atomicity: either every step completes, or the completed ones are compensated — never a silent partial mess.

Make it concrete with a booking: reserve a flight → reserve a hotel → charge the card → issue the ticket, each owned by a different service. The saga walks that chain, committing each step. If it gets all the way through — done. If a step fails partway, it runs the compensations for the completed steps, in reverse (cancel the hotel, cancel the flight). No coordinator holds anyone hostage; nobody blocks. But — and this is the whole rest of the lesson — that freedom has a price, and it starts with a word you can't take back: commit.

The first thing that trips people up: a compensation is not a rollback. A rollback un-does an uncommitted transaction — but every step of a saga has already committed (that's the point — it committed locally and released its lock). And you cannot un-commit a committed transaction (durability, ACID, Precisely). Watch what happens if you try, on real Postgres — commit a debit, then attempt to roll it back:

BEGIN; UPDATE accounts SET balance = balance - 100 WHERE name='alice'; COMMIT;  -- alice = 0
ROLLBACK;
-- WARNING:  there is no transaction in progress

There is nothing to roll back. The commit is final. So the only way to "undo" step 1 is a compensation — a brand-new forward transaction that reverses the effect:

BEGIN; UPDATE accounts SET balance = balance + 100 WHERE name='alice'; COMMIT;  -- restored

That distinction matters more than it looks. A compensation is a semantic undo, not a time-machine. Refunding a payment is not the same as the payment never happening — the customer saw the charge, the statement shows both lines, the fee may be non-refundable. And some effects can't be compensated at all: you can't un-send an email, un-ship a package, un-launch a rocket. Writing a saga is therefore two design jobs, not one: the forward step and its compensating step — and for each step you have to ask, honestly, can this actually be undone? When the answer is no, you have found something important, which the section after next is all about.

Here is the sharp edge. Because each step commits before the whole saga finishes, the intermediate state is visible to the entire world in between steps. A saga keeps three letters of ACID — A (all-or-compensated), C (consistency, eventually), D (durability) — but it throws away the I. There is no isolation.

We can watch the leak directly. Take a two-step saga that moves $100 from Alice to a merchant. Step one commits (Alice debited). Now, before step two, a concurrent observer reads the books:

observer sees:  alice = 0   merchant = 0   →   total = $0     (it was $100)

$100 has apparently vanished. For that window, anyone looking sees a world that violates your invariant. This is not a new problem — it's the dirty read from Isolation Levels & Their Anomalies, come back to haunt you at the saga level. And it's worse than a database dirty read, because these are committed rows: another saga running concurrently can read your half-done state and act on it (lost updates, fuzzy reads — the whole #46 menu returns).

You buy the isolation back with countermeasures, and the most important one is the semantic lock: a status column — PENDING → CONFIRMED / CANCELLED. A step sets the row PENDING; everyone is taught to only act on CONFIRMED rows; the final step flips it to CONFIRMED, and a compensation flips it to CANCELLED. It's an application-level lock that says "work in progress — don't touch." We added it and a well-behaved reader stopped seeing the ghost:

SELECT count(*) FROM orders WHERE status = 'CONFIRMED';   -->  0   (the PENDING order is hidden)

Other countermeasures round it out: commutative updates (design operations so order doesn't matter — a debit and its compensating credit commute), and the pessimistic view (re-order the saga so the riskiest, hardest-to-compensate step runs as late as possible, minimizing how long a dangerous intermediate state is exposed). The theme is the same: a saga has no isolation for free, so you engineer just enough of it back where the business actually cares.

We left a thread hanging: some steps can't be undone. That's not a flaw to patch — it's a property to design around, and it has a name. Every saga has a pivot transaction: the point of no return. It splits your steps into three kinds:

• Compensatable steps come before the pivot. They can be semantically undone, so if something later fails, you run backward recovery — compensate them in reverse and abort the saga.
• The pivot is the go/no-go commit — the moment the saga decides it's really happening. Once the pivot commits, there is no turning back.
• Retriable steps come after the pivot. They cannot be undone, so failure there is not handled by compensating — it's handled by forward recovery: you retry the step until it succeeds. Past the pivot, the only way out is through.

In the booking saga, charging the card is the natural pivot: reserving the flight and hotel are compensatable (release them), but once you've charged and committed to the trip, issuing the ticket is retriable — you don't un-charge a paid customer because the ticket printer jammed; you keep trying to issue the ticket. Getting this classification right is the crux of saga design, and getting it wrong is a classic outage: if you mislabel a step as compensatable when it really can't be reversed, your "recovery" tries to undo something that has already left the building, and you're left with orphaned, half-applied state. So before you write a line of it, sort every step into compensatable, pivot, or retriable — and put the pivot where the business truly commits.

The pivot changes the direction of recovery, which is exactly the kind of thing you should watch happen rather than read about. Below is the booking saga as a live chain. Run it once clean, then pick a step to fail.

Fail the flight or the hotel — before the pivot — and watch the saga run its compensations backward: each completed step flips to cancelled, in reverse order (and remember, each of those is a new transaction, not a rollback). Then fail the ticket — after the pivot — and watch the whole recovery flip the other way: the charge has already gone through and can't be undone, so the saga refuses to abort and instead retries forward until the ticket issues. Same failure, opposite response, and the pivot is the hinge. Try each one until the rule is in your fingers: before the pivot, undo; after the pivot, push through.

One question is left: who runs the saga? — who calls step two after step one, and who fires the compensations when something breaks? There are two answers, and it's a real architectural fork.

Orchestration puts a single orchestrator in charge: a central component that holds the workflow, calls each service in turn, and triggers the compensations on failure. The logic of "what happens next" lives in one place, which makes the saga easy to understand, monitor, test, and reason about — you can point at the orchestrator and see the whole flow. The cost is that central component: it's one more thing to build, run, and keep available, and it can become a bottleneck.

Choreography removes the central brain entirely. Each service just emits an event when it finishes its step ("flight reserved"), and the next service listens and reacts. It's beautifully decoupled — services don't know about each other, deploy independently, and scale well — but the workflow now exists nowhere and everywhere: the business logic is smeared across services and a web of events, which makes it hard to trace, hard to see the whole picture, and hard to be sure every failure path actually triggers its compensation. (It also leans entirely on events being delivered reliably — a dropped event silently strands the saga — which is a problem the next lesson exists to solve.)

Neither wins outright. The common, sensible answer is both: choreography for simple, high-throughput, loosely-coupled flows, and orchestration for the complex, stateful, business-critical workflows where you need to see and control what's happening.

Put it back in the frame of the three moves. Avoid the distributed transaction if you can (collapse to one owner). If you can't, you're choosing between coordinate (2PC — atomic and isolated, but it blocks) and compensate (saga — never blocks, stays available, but isn't isolated). Reach for a saga when you have a genuinely multi-owner, long-running business process, you cannot afford to block (holding locks across services for the whole flow is a non-starter), and you can tolerate the temporary inconsistency and pay for the countermeasures. That's most real microservice workflows — which is why the saga is the workhorse of distributed data consistency.

The takeaways:

• A saga is the "compensate" move: a chain of local commits, each releasing its lock immediately, plus a compensation for each step. Either all complete, or the completed ones are compensated in reverse. No global lock, no coordinator, no blocking — the opposite of 2PC.
• Compensation isn't rollback. You can't un-commit a committed step (proven: WARNING: there is no transaction in progress); a compensation is a new forward transaction that semantically reverses it — and some effects can't be reversed at all.
• The price is isolation. A saga is ACD, not ACID — the intermediate state is visible (measured: mid-saga an observer sees total = $0), so the #46 anomalies return. Buy it back with a semantic lock (PENDING → CONFIRMED / CANCELLED), commutative updates, and the pessimistic view. Every step and compensation must be idempotent (measured: a retried compensation goes UPDATE 1 then UPDATE 0).
• The pivot decides recovery direction. Classify steps compensatable → pivot → retriable: fail before the pivot → compensate backward; fail after → retry forward. Misplacing the pivot leaves orphaned state.
• Run it as orchestration (central, easy to reason about) or choreography (event-driven, decoupled but hard to trace) — usually both. Next — Outbox + CDC: Reliable Events from Your Database: choreography (and every saga step) depends on events being delivered without loss — which runs straight into the dual-write problem from the start of this section, and the pattern that finally solves it.