ZooKeeper & etcd in Practice

The last lesson ended with a spec: "You now know what a coordination service must be on the inside: a consensus core, sessions and leases at the rim, a token fountain for the honest resources." This lesson is about what it's like to sit in the driver's seat.

Here's the deal these systems offer, and it's one of the best deals in engineering. Consensus is the hardest problem this course has touched: seven lessons of quorums, terms, logs, and ballots hard. ZooKeeper and etcd wrap all of it in a small API and run it as a utility: a handful of nodes, replicated by a consensus core, holding tiny amounts of load-bearing data on behalf of every other system you operate. Your services don't run Raft. They ask a service that does.

What do you buy with that? Not storage (these things hold megabytes, not terabytes). You buy agreement with an API: who holds the lock, who is the leader, what the current config is, which workers exist right now. The famous trick of the trade is that you don't program these answers directly; you compose them out of a few odd-looking primitives, following recipes that the field has hammered out over fifteen years. Lock, election, notice board: three recipes cover most of what production needs.

This lesson teaches the primitives, the three recipes, and the one classic mistake — the herd effect — whose fix is the single cleverest line in the cookbook. Then it walks the fence line at the edge of everything this section assumed, and points at the strange country on the other side.

Underneath, both systems are what the last lesson said all serious lock services are: a consensus cluster wearing an API. etcd is Raft, which you can now read fluently. ZooKeeper predates Raft and runs its own protocol, ZAB, a different dialect of the same family, with a leader, majorities, and a totally ordered log of updates. For everything in this lesson, the difference doesn't matter; that's rather the point of buying coordination as a service.

What does matter is the shape each one hands you.

ZooKeeper speaks in a tree. Its world is znodes: tiny named nodes in a filesystem-ish hierarchy, each holding a few bytes. Three modifiers do all the work. A znode can be ephemeral: tied to your session, it evaporates the instant your session dies — presence made physical, the lease from last lesson wearing a filename. It can be sequential: the server appends a monotonically rising number to the name, lock-0001, lock-0002. Yes, that's a token fountain. And any znode can carry a watch, but read the docs' sentence carefully, because it's the most misread line in the system: "A watch event is one-time trigger." Not a subscription. It fires once, then you must look again and re-arm, and the recipes live and die on getting that loop right.

etcd speaks in a ledger. A flat key-value space where every change bumps a global revision: the whole store is versioned like one long diary, and each key remembers the revision that last touched it. Leases are first-class (you met them attached to keys last lesson); watches are streams, not one-shot triggers: subscribe to a key or a whole prefix and change events flow until you hang up; and transactions are a tiny atomic if-then-else: if this key's revision is still 41, then write this, else tell me — compare-and-swap, the primitive every safe recipe leans on.

And here is the detail that should make the previous lesson light up: the revision is a fencing token. It's a number, it only goes up, and the store mints it for free with every grant. When analysts formally checked etcd's locks, that was exactly the guidance: pair the lock with its revision and make the resource check it. ZooKeeper's transaction counter, the zxid you met guarding split-brain, plays the same role. The token fountain isn't an add-on you build; it's built in, waiting to be used.

Now the flagship recipe, built the wrong way first, because the wrong way is what everyone writes on day one, and its failure teaches the design language.

The obvious lock: every client tries to create the same ephemeral znode, /app/lock. One succeeds and holds; the rest set a watch on it and wait. Holder finishes (or dies: ephemeral, so death releases), the znode vanishes, watches fire, everyone retries. Correct? Actually yes: mutual exclusion holds. So run it with fifty waiting clients and watch the bill instead: the lock releases, and the coordinator fires fifty notifications at once. Fifty clients wake, fifty stampede the cluster with create requests, forty-nine lose and set watches again, and this repeats on every single handoff. That's the herd effect: a thundering herd purchased at the coordination layer, the most load-bearing place you own. The lock works; the system around it degrades.

The recipe fixes it with a line. Instead of everyone fighting over one name, each client creates an ephemeral sequential znode under the lock's directory: the server names them lock-0001, lock-0002, lock-0003… Now the rule: the lowest number holds the lock, and — the clever line — each waiter watches only its immediate predecessor, the node one number below its own. (The official recipe even specifies listing the children without a watch, precisely to avoid re-creating the herd.)

Play the failure through it. The holder, lock-0001, finishes or dies; its znode vanishes; exactly one watch fires — lock-0002's; one client wakes, confirms it's now lowest, and holds. Fifty waiters, one notification, zero stampede. Kill a client in the middle of the line and it's even prettier: lock-0003's predecessor evaporates, one watch fires, lock-0003 looks, sees lock-0002 still ahead, and quietly re-watches it. The line heals around the corpse and the holder never hears a thing.

Two footnotes from the last lesson, still binding. The queue orders claims, not truth: a paused holder is still a paused holder, so pair the lock with the built-in token (zxid, revision) at any resource that can check. And don't hand-roll this: the edge cases (connection lost mid-create: did my node get made?; watch re-arm races) are exactly why hardened implementations exist: Curator's recipes for ZooKeeper, the concurrency package for etcd. Read the recipe to understand it; run the library.

Here's the payoff for learning the lock properly: leader election is the same recipe. Every candidate creates an ephemeral sequential znode; the lowest number is the leader; everyone else watches their predecessor and waits their turn. That's it. The election lesson was randomized timers and vote-splitting because those nodes had to elect without a referee, but your services have a referee now, one that already paid the consensus bill. On this side of the API, an election is just a queue with a crown on the front.

Look at what the recipe gives you for free. Succession is orderly: when the leader's session dies, exactly one successor is notified: no campaign, no split votes, no storm; the line simply advances. The zombie is handled the same way as ever: a paused ex-leader that wakes still believing must be fenced at the resources, and the coordinator's rising numbers are right there for the checking. And you've already operated this: Patroni holding its leader key in a consensus store, demoting the Postgres whose lease expired: that's this recipe, in production, deciding who gets to be your primary.

One honest caution carried over from the heartbeats lesson: the coordinator judges clients by session liveness, which is a timeout: the same failure detector with the same pause-shaped blind spot. A leader that freezes long enough loses its session and its crown while still believing it reigns. The recipe doesn't repeal the pause; it just makes the succession that follows it clean.

The third recipe barely looks like a recipe, which is why it quietly runs more infrastructure than the other two combined.

Configuration: put the settings that must be agreed upon — feature flags, the current schema version, the address of the primary — into znodes or keys, and have every service watch them. Change the value once, and the watch machinery pushes the news to every subscriber: a config rollout with no deploy, no polling loop, and no version skew you didn't choose. (Mind the dialects: in ZooKeeper you must re-arm after every one-shot trigger; the classic bug is acting on the event and forgetting to look again, missing the next change. etcd's streaming watches carry that burden for you.)

Presence and discovery: each worker registers an ephemeral node — /workers/w-17 — that exists exactly as long as its session does. The list of children is the live membership, maintained by nobody: a worker that dies is un-registered by its own evaporating znode. Watch the directory and you know, within a session timeout, who's alive.

If that sounds like it competes with the gossip lesson, sharpen the boundary, because this is a favorite interview probe. Gossip-style membership scales to thousands of nodes by being cheap and approximate; the division of labor was "consensus for the decisions, gossip for the news." The notice board is the decisions side: strongly consistent presence for the things that must be exact — the set of shard owners, the one current primary, the config everyone must agree on, paid for with the coordinator's consensus writes, which is why it's the right tool for dozens of load-bearing entries and the wrong one for ten thousand chatty ones.

The herd effect is a thing you should count, not just read about. The lab below is the waiting line with both watch strategies and a scoreboard.

Run the comparison:

1. Build a line. Add five or six clients and read the znode queue: lock-0001 holds, the rest wait in server-stamped order.
2. Stampede first. In everyone-watches-the-lock mode, release the holder. Count what flies: a notification to every waiter, a wave of futile grab-attempts back, one winner. Release again: same storm, one client fewer. The counters are keeping score of pure waste.
3. Now the recipe. Switch to watch-your-predecessor, rebuild the line, release. One notification. One wakeup. Same winner. The scoreboard difference is the recipe's whole argument.
4. Kill the middle of the line. Click a waiter mid-queue: its session dies, its ephemeral node evaporates, and exactly one neighbor re-wires its watch — the holder never notices, and in naive mode compare the same click's blast radius.
5. Kill the holder. Death is release: the ephemeral lock is the lease made visible, and the line advances with the discipline the mode you chose deserves.

Watch arrows are drawn live in both modes: a star of watches all aimed at one node, versus a quiet chain where each client minds exactly one neighbor. Every recipe in this lesson is some arrangement of that arrow.

Advice for the day this stops being theory, then one paragraph of geography.

Run an odd, small ensemble. Three or five nodes: majorities, exactly as the quorum lesson priced them: five tolerates two deaths, and even sizes buy risk, not safety. Spread them across failure domains; five coordinators in one rack is one power supply away from being zero.

Respect it as your most load-bearing dependency. When the coordinator is down, nobody elects, nobody locks, config freezes. Design your services to degrade politely in that window: keep working off the last-known config, keep serving with the leader you already have, alarm loudly, and never make request-path latency depend on a coordinator round-trip. The coordination service is where decisions live, not where traffic goes.

Remember the pause. Session timeouts are heartbeat verdicts, and the coordinator is as blind as any detector from the failure-detection lesson: a long GC pause looks exactly like death, the session dies, ephemeral nodes evaporate, crowns move. That isn't a bug to file; it's the previous lesson's whole argument arriving in production: expect evictions you didn't deserve, and keep the fencing checks armed for the zombie that follows.

And the map's edge. Every protocol in this section — Paxos, Raft, ZAB, every quorum and every recipe — assumes nodes are honest but mortal: they crash, they pause, they lose mail, but they never lie. Drop that assumption (nodes that send contradictory messages, compromised or arbitrarily buggy) and you've crossed into different country: Byzantine fault tolerance, with its own protocols (PBFT and kin) that pay roughly triple the replicas for the privilege, and the blockchain world's Nakamoto consensus, which buys honesty with proof-of-work instead of trust. Real, rigorous, and out of this course's scope. Infrastructure inside one organization almost always earns the honest-nodes assumption instead: signed and checksummed messages, so corruption downgrades into a crash you already know how to survive. You watched that exact move in the failure-models lesson — S3 didn't deploy Byzantine consensus after the bit-flip; they added checksums to the gossip. Armor at the border, and stay on the honest side of the map.

• Coordination is a service you buy, not a protocol you write. A small consensus core (Raft in etcd, ZAB in ZooKeeper: same family, different dialect) holding megabytes of load-bearing agreement for everything else you run.
• Learn the primitives, compose the recipes. ZooKeeper: znodes, ephemeral (presence that evaporates with the session), sequential (a built-in number fountain), plus one-shot watches you must re-arm. etcd: revisions, leases, streaming watches, and txn's atomic if-then-else.
• The waiting line is THE recipe. Ephemeral sequential nodes, lowest holds, each waiter watches only its predecessor. One release, one notification — the herd effect is the tax on getting that one arrow wrong.
• Election is the lock wearing a crown. Lowest number leads, succession is the line advancing, and Patroni has been showing you this recipe since the failover codelab.
• The notice board is the decisions side of the news. Watched config and ephemeral presence: strongly consistent, push-based, priced for dozens of entries; gossip still owns the ten-thousand-node crowd.
• The tokens are already minted. zxid and the etcd revision are fencing tokens the store gives you for free. Pair every correctness lock with one.
• Ops: odd ensembles across failure domains; degrade politely when the coordinator blinks; sessions are timeouts, so the pause still collects its tax. Use Curator and the concurrency package — the recipes' edge cases have already eaten their teams.
• The map's edge: everything here assumes honest-but-mortal nodes. Liars live across the border with PBFT and Nakamoto; checksums are the passport that keeps you home.

You've now read every recipe this section has to offer. What's left is the thing reading can't give you: the feel of it under your hands — a real cluster, a real election, real sessions dying in your terminal while you watch the crown move. Next: Codelab: Leader Election with etcd.