Codelab: Leader Election with etcd

Eight lessons of theory end here, at a keyboard. In the next twenty minutes you will run a real coordination service, win a real election, queue behind a leader, watch succession happen two different ways (one instant, one painfully honest), and then drop below the floorboards to the Raft cluster underneath, where you'll kill an actual consensus leader and watch a term change, then kill a majority and watch the whole machine choose silence over a second history.

Nothing here is simulated and nothing needs code: one brew install, four terminal windows, and every command below was run exactly as printed (etcd 3.7.0, macOS): the outputs you see are captured, not imagined. The cast: etcd, the coordination service from the last lesson, and etcdctl, its command-line client, which ships with a complete leader election built in. You already know every idea you're about to type. That's the point.

You need etcd and its client. On macOS with Homebrew (Linux: your package manager has etcd, or grab the release binaries from etcd's site):

Any 3.4+ version behaves the same for everything below. Open four terminal windows and arrange them so you can see all four at once. This lab is a choreography, and half the fun is watching one terminal react to what you typed in another. We'll call them T1 (the server), T2 (alice), T3 (bob), and T4 (your X-ray and murder weapon).

In T1, start a single etcd. It logs a few dozen JSON lines and then sits there: that's a healthy coordination service with nobody to coordinate yet:

In T4, prove it's alive with the notice-board primitive from last lesson, a key, written and read:

etcdctl talks to localhost:2379 by default, which is exactly where your server is listening. One node is a legitimate one-member Raft cluster (a parliament of one), useful for everything in Act 1, and exactly what we'll tear up in Act 2.

In T2, run for office. The tool asks for an election name and your proposal (any value; traditionally, who you are):

Two lines print, and then the command hangs. Both facts are the lesson. The first line is alice's campaign key, the election name plus her session's lease ID; the second is her proposal, now readable by anyone. And the hang isn't a bug: holding the leadership is a full-time job. Behind that frozen prompt, the client is keeping a session lease alive: the watchdog heartbeat from the locks lesson, running right in front of you. The moment that process stops renewing, by any means, the crown is forfeit. Leave it hanging; it's reigning.

In T3, run bob for the same office:

Silence, indefinitely. bob created his own campaign key and is now blocked, watching, waiting his turn. It's the waiting line from the recipes lesson, except now it's a real process in a real terminal refusing to give you your prompt back. No polling, no retries, no stampede: bob will say nothing until the moment he legally holds the election.

Nothing about that queue is hidden. In T4, read the election's keys like any other data:

Two campaign keys, one per candidate. Now ask for the detail that decides everything:

There it is — the whole theory of the last two lessons in four lines of output. Each candidate holds a key stamped with a creation revision, the store's rising number. alice's is 3, bob's is 4, and lowest revision leads — the waiting line, implemented exactly as the recipe said, ordered by the same number fountain the fencing lesson told you to hand to your resources. You are looking at a fencing token with a campaign attached.

One more window into the machine. In T4, watch the election as a stream:

The -l flag means listen: print the current leader, then stream every future one. This is the notice board watching itself, the exact primitive your services would use to learn who their primary is without asking anyone. Leave it running; it's about to narrate a transfer of power.

Go to T2, where alice reigns, and press Ctrl-C. Now look at the other terminals, in the same second:

Instant. No timeout, no drama. A clean exit gives the client a chance to revoke its own lease on the way out — resignation, not death — so alice's campaign key vanished immediately, bob's key became the lowest revision standing, and his blocked command woke up holding the crown. One release, one notification, and the line advanced. You've watched this in three simulators; it's better in person.

Now the difference the whole locks lesson was about. Re-enter alice in T2; she queues behind bob, exactly as bob once queued behind her:

This time, don't resign bob. Murder him. In T4 (Ctrl-C the observer first), kill the process with no chance to clean up:

Sit with this minute, because it's the most honest minute in the section. bob's process is gone, but etcd doesn't know that. All it knows is silence, and you know what silence means: dead, slow, or unreachable, no one can say. So the coordination service does the only defensible thing: it waits out bob's session lease — for etcdctl elect, sixty seconds by default — and only when the lease expires does bob's key evaporate and alice's terminal spring to life. Time it. That long, awkward wait is not a flaw; it's the lease-versus-pause trade-off priced in real seconds. Clean exits resign instantly; dirty deaths cost a lease. (And if bob had been merely paused, not dead, he'd wake believing he still led — which is why the fencing lesson taught your resources to check the rising number you X-rayed in Step 4.)

Everything so far ran on a parliament of one. But the last lesson's whole claim was that a coordination service is a consensus cluster wearing an API — so let's wear the API less and see the cluster. Stop everything from Act 1 (Ctrl-C in every terminal, then rm -rf ./one-node). Now start a real three-member cluster: T1, T2, T3 each become an etcd member. The flags are verbose but mechanical — each member states its own address and the full roster:

Give them five seconds to hold their election (you know exactly what they're doing in there). Then, in T4, ask each member who it thinks rules:

One member says IS LEADER: true and all three agree on the term. That's not the recipe from Act 1 — that's Raft itself, the randomized timers and majority ballots from the election lessons, which just ran under your fingers in half a second. Write a key so there's something to protect: etcdctl --endpoints=127.0.0.1:2379 put color blue.

Find the terminal whose member said true and press Ctrl-C on it. Then ask the survivors:

Read those two changes like the graduate you now are: a new leader and a term bumped from 2 to 3. A timer fired, a candidate with a complete log won a majority, and the reign changed — the entire Raft lesson, executed in the time it took you to switch windows. And color is still blue: committed means on a majority, and a majority survived. Two of three is enough. Which raises the obvious next question.

Ctrl-C a second member. One of three remains. Try to write:

The survivor is healthy, reachable, and holds all the data — and it refuses to decide anything. One of three cannot know whether the other two are dead or merely partitioned away taking writes of their own, and you proved with pigeonholes why guessing is forbidden: one answer never taken back requires a majority, and a majority of three is two. The cluster chooses silence over a second history. Every lesson in this section is in that one error line. Now resurrect any dead member (press ↑ and Enter in its terminal) and try again:

Two of three alive, majority restored, decisions flow. The parliament is back in session.

Walk back through your history; each command was a lesson from this section wearing work clothes.

• elect blocked and held — a session lease kept alive by a watchdog heartbeat: the lease from Distributed Locks & Leases, doing its full-time job.
• bob's silent queue — the waiting line from ZooKeeper & etcd in Practice: one campaign key each, lowest wins, no stampede when power changed hands.
• CreateRevision: 3 and 4 — the store's rising number ordering the line: the token fountain, and the fencing token your resources should demand.
• Ctrl-C versus kill -9 — resignation revokes the lease instantly; murder makes the service wait out sixty honest seconds of silence, because dead, slow, and unreachable still look identical. The failure-detection bet, priced in wall-clock time you personally sat through.
• IS LEADER and RAFT TERM: 2 → 3 — Raft's election, run live when you killed a member: randomized timers, majority ballots, a new reign with a higher number.
• context deadline exceeded with one of three alive — the pigeonhole from Quorum Intuition, Revisited: no majority, no decisions, silence over a second history.
• The observer stream — a watch on the decisions side of the notice board: consensus for the decisions, gossip for the news.

• The election recipe is three commands. elect <name> <proposal> to campaign (and block, and reign), the same command to queue, -l to observe. Everything else is the machinery you spent a section learning, doing its job.
• Blocking is holding. A leader is a process that keeps renewing a lease; the frozen prompt is the heartbeat. Kill the renewals and the crown moves — instantly if you resign, one lease-length later if you die rudely.
• The internals are just keys. Campaign keys under a prefix, ordered by creation revision — you can get --prefix an election like any other data, and the revision that orders it is the fencing token your storage should check.
• Beneath every recipe, a parliament. Three members, one IS LEADER, terms that bump when you kill kings, and a hard stop the moment a majority is gone. You watched the abstraction and its implementation in the same afternoon, and they told the same story.

That's the section — every promise from The Consensus Problem onward, cashed in a terminal. What remains is to prove you own it: ten decisions, drawn from everything between FLP and the herd effect, in Checkpoint: Consensus & Coordination.