Read Repair & Hinted Handoff

I have been lying to you for four lessons.

Not about the results — every one of those was real. But every single experiment in Quorums: N, R, W (#23), The Consistency Spectrum (#24), CAP, Properly (#25) and PACELC: CAP's Grown-Up Sibling (#26) was run on a cluster I had deliberately crippled. In every one of them, the first thing I did was this:

I turned off Cassandra's immune system, on purpose, so the wounds would stay open long enough for you to look at them. Stale reads that never healed. A divergence that sat there for twelve straight reads. Ana's hat, silently eaten and never recovered.

A real cluster does not behave like that. A real cluster fixes itself, quietly, constantly, while nobody is watching — and the machinery that does it is genuinely elegant.

So today I switch it all back on.

And then I am going to show you the thing that made me sit back from my keyboard: I deleted a user, and she came back from the dead. Not through a bug. Not through data corruption. She was resurrected by the healing machinery itself, working exactly as designed.

Here's the elegant idea. You are already reading from multiple replicas — that's what a quorum read is. So while the coordinator is in there anyway, comparing answers… why not fix what it finds?

That's read repair, and here is how it actually runs, which is more careful than people assume.

The coordinator does not ask three replicas for the full row — that would be a waste of network. It asks one replica for the data, and asks the others only for a digest: a hash of what they hold.

"A digest read request is not a full read and only returns the hash value of the data."

If the hashes agree, everyone's telling the same story, and the coordinator hands you the answer. Done — no extra cost.

If the hashes disagree — a digest mismatch — the coordinator now knows something is wrong and goes and gets the full data from everybody. Then it reconciles:

"Data from the two replicas is compared and based on the timestamps the most recent replica is selected."

…and — this is the bit that matters — it writes the newest value back to the replicas that were behind. It doesn't just answer you correctly. It leaves the cluster better than it found it.

So I turned it back on and watched, reading each replica's own SSTables directly with sstabledump — because if I ask the cluster, the coordinator will reconcile the answer for me and hide exactly what I'm trying to see. I want to know what each machine physically has on its disk.

Nobody wrote to rr3. Nobody repaired it. Somebody merely looked at that row, and the act of looking fixed it.

That is a genuinely lovely piece of engineering, and there's a bonus in it that pays off a debt from two lessons ago. Cassandra's default is blocking read repair — meaning the read does not return to you until the repair writes have landed. Why be so strict? Because of this:

"Cassandra uses a blocking read repair to ensure the expectation of 'monotonic quorum reads' — i.e. that in 2 successive quorum reads, it's guaranteed the 2nd one won't get something older than the 1st one."

Monotonic reads. In The Consistency Spectrum (#24) I showed you a user refreshing a page twice and watching an edit disappear — time running backwards — and I told you the cheap fix was a sticky session. This is the other fix, and it's the one the database performs on your behalf. By dragging the lagging replica forward before answering you, a blocking read repair makes it impossible for your next read to fall behind this one.

The anomaly in #24 and the mechanism in #27 are the same story, told from opposite ends.

Read repair fixes things on the read path. But there's an obvious gap: what about a write that arrives while a replica is already down? Nobody's reading that row. Nobody may read it for weeks. Is it just… lost?

No — and the fix is charmingly low-tech. When the coordinator tries to write to a replica and finds it dead, it writes itself a note.

"coordinators attempting to write to those replicas store temporary hints on their local filesystem for later application."

That note — a hint — contains the target node and the whole mutation, serialised as a blob. It sits on the coordinator's disk. And when the dead node comes back, the hints are streamed to it and replayed, and it quietly catches up on everything it missed.

The lovely thing is that this happens with no read at all. Nobody has to look at the data. I tested it exactly that way — turned hints on, killed a node, wrote to it, brought it back, and then deliberately never issued a single read:

So now we have a tidy little story, and it's the story most engineers carry around in their heads:

• Hinted handoff catches the writes that arrive while a node is down.
• Read repair catches whatever slips through, as soon as anyone reads it.
• Between them, the cluster heals itself. That's what "eventual consistency" means.

It's a good story. It is also wrong, and the rest of this lesson is about how.

Read repair heals a row when you read it. Which raises a question that nobody seems to ask out loud:

What happens to the rows nobody reads?

So I measured it. Six keys. I took a replica down, updated all six while it was gone, and brought it back — so all six are stale on that node. Then I read exactly one of them, and went and looked at the node's disk.

One in six.

Read repair only ever heals the rows you actually read. And most of your data — your cold archive, your 2019 orders, the audit log, the user who hasn't logged in since March — nobody ever reads it. It is sitting on some replica, wrong, and it will sit there wrong forever. There is no background process coming for it. Nothing is going to notice.

And it's actually narrower than that, in a way I didn't expect until I measured it. Look again at what the docs say read repair covers:

"If read repair is performed it is made only on the replicas that are not up-to-date and that are involved in the read request."

Involved in the read request. A QUORUM read on three replicas talks to two of them. So if the coordinator happens to pick the two replicas that were already fine, it never asks the stale one — sees no mismatch — and repairs nothing. I caught this live:

So even reading a row doesn't guarantee it gets healed. It only gets healed if the read happened to touch the broken replica.

A read is not a repair. A read is a lottery ticket for a repair.

At this point you might reasonably think I'm being unfair — nitpicking at mechanisms that are obviously doing their best. So let me hand the microphone to Cassandra's own documentation, which is refreshingly, brutally honest about all of this. It says the quiet part out loud, twice:

"Hints are best effort, however, and do not guarantee eventual consistency like anti-entropy repair does."

"Hints, like read-repair, are best effort and not an alternative to performing full repair."

Read that again, slowly, because it's remarkable.

The two mechanisms that everybody names when you ask "how does eventual consistency actually work?" — the two mechanisms this lesson is named after — are, by the vendor's own admission, best effort, and they do not guarantee eventual consistency.

And hints have a hard limit that I want you to remember, because it will bite you at 3am. A hint is kept for max_hint_window, which defaults to three hours. What happens if your node is down for three hours and one minute?

"If the node does not return in time, the destination replica will be permanently out of sync until either read-repair or full/incremental anti-entropy repair propagates the mutation."

Permanently. A node that was down for a long weekend comes back with holes in it — holes that nothing in the system is ever going to close by itself.

So: hints cover one write, for three hours. Read repair covers rows you read, on replicas that read happened to touch. Neither guarantees anything.

Which means there has to be a third thing. And there is.

It's called anti-entropy repair, and you run it by typing nodetool repair.

It works by comparing Merkle trees — "a hierarchy of hashes". Each node hashes its data in chunks, then hashes the hashes, all the way up to a single root. Two nodes compare roots: if they match, the entire dataset is identical and you're done — one hash comparison, no data moved. If they differ, you descend into the children and repeat, narrowing in on exactly which ranges disagree. You find the divergent rows without shipping the dataset across the network.

And unlike the other two, it doesn't care whether anybody read the data, or whether a hint survived, or whether the right replica happened to be in the read set. It just goes and compares everything.

I ran it on the cluster with four rotting keys:

It fixed the data nobody looked at. It is the only mechanism that can.

So why isn't this the hero of the story? Why does nobody talk about it at conferences?

Because of one sentence in the docs, and it is the most consequential sentence in this entire lesson:

"repair can result in a lot of disk and network io, it's not run automatically by Cassandra. It is run by the operator via nodetool."

The one mechanism that actually guarantees convergence is the one your database will never run for you. It's expensive, so they made it your problem. It lives in your crontab, or it doesn't live at all.

And I promise you this: on a depressing number of production Cassandra clusters, it doesn't live at all. Somebody set it up in year one, it got noisy, it got disabled during an incident, and it never came back.

Which brings us to what happens then.

To understand this you need one fact about deletes that surprises almost everybody the first time they hear it:

A DELETE does not delete anything. A delete is a write.

In a leaderless system you cannot go around erasing rows, because you can't reach all the replicas — some of them are down, that's the whole premise. And if you did erase a row, a replica that still had it would look at the others, see nothing, conclude they must be missing it, and helpfully replicate it back.

So instead, a delete writes a tombstone: a small marker that says "this row is dead, as of timestamp T." It's a gravestone. It doesn't remove the body; it records the death. And because it has a timestamp, it beats any older version of the row — so it acts like a delete.

But gravestones take up space. A database that never forgot its dead would fill up with nothing but gravestones. So tombstones themselves eventually get cleaned away, after a grace period called gc_grace_seconds — which defaults to ten days.

Ten days is a long time. Long enough for everything to be fine.

Unless it isn't. Here is exactly what I ran.

She's back. On every replica. Permanently.

Follow the logic of step 5, because it's the whole horror. Two replicas have nothing. One replica has a row. And there is no longer any evidence anywhere in the cluster that this row was ever deleted — the tombstone, the only thing that remembered, was swept up as garbage.

So what is the cluster supposed to conclude? It sees two nodes missing a row that a third node has. That looks exactly like the situation read repair exists to fix. So it does its job. It fixes it. It resurrects her.

Something beats nothing. That's the whole rule, and it's normally a good rule — it's how a replica that missed a write gets caught up. But once the tombstone is gone, a deleted row and a missed row look identical, and the rule that usually saves you turns around and bites.

You did not lose data. You un-deleted it. And the mechanism that undid your delete was the healing machinery, working perfectly, exactly as designed.

No error. No alert. Nothing in the logs in red. The user who asked to be forgotten is back in your database, and the first you'll hear about it is when she emails you again — or when a regulator does.

One more detail from the run, because it's the nastiest part. My first read didn't resurrect her:

a QUORUM read  ->  (0 rows).   She stayed dead.

She stayed dead by luck. That read happened to ask the two empty replicas and never touched the one holding the body. The resurrection is a coin flip on any single read — and a certainty over time. The next read, or the one after, will touch that replica. And the instant one does, read repair makes the resurrection permanent, everywhere.

This isn't a theoretical hazard I dressed up. It has a name in the docs, and the rule to prevent it is stated plainly:

"repair should be run often enough that the gc grace period never expires on unrepaired data. Otherwise, deleted data could reappear."

With the ten-day default, the recommendation is to repair every node at least once every seven days. Seven days. That's the margin between you and the dead rising.

This lesson has a lot of moving parts, and the only way to really own them is to run the cluster into the ground yourself.

Below is a three-replica cluster holding six keys. You can kill and revive nodes, write, delete, and — crucially — read one specific key at a time. Every replica's actual on-disk state is shown, the way sstabledump showed me: green for correct, amber for stale, and a gravestone for a tombstone.

Try this first. Kill a node. Write to all six keys. Bring it back. Now read one key and watch that single cell go green — and watch the other five stay amber. That's the coverage gap, in your hand. Read a second key. A third. You are healing your cluster one row at a time, by hand, and you will get bored long before you're done. Now press repair, and watch the whole grid go green at once.

Then go and do the murder. Turn hints off, kill a node, delete a key, let the tombstone expire, and bring the node back. Read the key. Watch it come back from the dead — and then watch the healing machinery spread the corpse across the whole cluster.

Try to prevent it. You have exactly one tool that works, and it isn't either of the two this lesson is named after.

Go and check whether repair is scheduled. Right now, before you read the next paragraph. Not "is repair possible" — is it scheduled, is it running, and did it succeed last week? This is the single highest-value thing in this lesson, and it takes ten minutes. The usual answer in the wild is "we set it up once and then it started failing and someone turned it off."

Know your two numbers, and know which is bigger. gc_grace_seconds (default: 10 days) is how long your tombstones survive. Your repair interval is how long it takes you to notice a divergence. If your repair interval is longer than your gc_grace, you are running a zombie farm. The recommended margin is repair every 7 days against the 10-day default.

Treat a node that was down for more than three hours as damaged. Not "recovered" — damaged. Its hints have expired; it has holes; and only a repair will close them. max_hint_window is a three-hour promise, and the world does not care about your promise.

Stop saying "it's eventually consistent" as though that were a property of the software. It is a property of your operations. The database offers you two best-effort patches and one real guarantee that you have to schedule yourself. If nobody schedules it, the "eventually" never arrives — and one day the dead will walk.

And if you have deletion obligations — GDPR, right-to-be-forgotten, retention policies — this is not a performance concern. It is a compliance one. A resurrected user is not a stale cache. It is a record you told a regulator you had destroyed, sitting in your primary datastore, restored by your own database.

"Eventual consistency means the database heals itself." It means the database can heal itself if somebody schedules it. The healing you get for free is best-effort and partial. "Eventual" is not a property your database has — it is a chore somebody has to do.

"Read repair keeps my cluster consistent." It repairs only rows you read, and only on replicas that read happened to touch. Measured: one key of six healed — and a QUORUM read of a stale key healed nothing, because it never asked the stale replica. A read is a lottery ticket for a repair, not a repair.

"Hinted handoff means no write is ever lost." Hints are best effort, they sit on the coordinator's local disk, and they expire after three hours by default. After that, the docs say the replica is "permanently out of sync."

"A DELETE deletes." A delete is a write. It writes a tombstone — a gravestone, not an erasure. And tombstones expire.

"Repair is what you do when something has gone wrong." Repair is what you do on a schedule so that nothing goes wrong. It's not incident response. It's dental hygiene.

"If my reads come back correct, my replicas must be fine." No. The coordinator reconciles the answer for you, on the fly. A correct answer tells you nothing about the state of your replicas. For four whole lessons I got perfectly sensible answers out of a cluster whose replicas were openly contradicting each other.

"Deleted data is gone." Deleted data is gone if every replica saw the tombstone before it expired. Otherwise it is a coin flip, and eventually the coin comes up heads.

• Read repair heals on the read path: digest hashes → mismatch → fetch full data → reconcile by timestamp → write the newest value back. Measured: a stale replica healed on disk from a single read, with nobody writing to it.
• Blocking read repair is what buys you monotonic reads — the anomaly from The Consistency Spectrum (#24), where time ran backwards. The database drags the lagging replica forward before answering you.
• Hinted handoff heals on the write path: the coordinator writes itself a note and replays it when the node returns. Measured: a node healed with zero reads. But the note is only kept for three hours.
• ⭐ Neither one guarantees anything, and the vendor says so: "Hints, like read-repair, are best effort and not an alternative to performing full repair." Read repair covers only rows you read, on replicas that read touched — I measured one key of six healed, with the other five left to rot.
• Anti-entropy repair (nodetool repair) is the only mechanism that actually guarantees convergence. It compares Merkle trees and fixes the data nobody looked at. And "it's not run automatically by Cassandra — it is run by the operator." It lives in your crontab, or it doesn't live at all.
• ⭐⭐ A DELETE is a write — a tombstone. Tombstones expire (gc_grace_seconds, default 10 days). If a replica misses the tombstone and comes back after it expired, there is no longer any evidence the row was ever deleted — so the healing machinery resurrects it, permanently, across the whole cluster. I deleted a user and she came back. Repair every node at least every 7 days, or you are running a zombie farm.

And notice the sentence we walked straight past. When read repair found a mismatch, it decided the winner "based on the timestamps — the most recent replica is selected." Most recent according to whom? Those timestamps come from clocks, on different machines, that disagree with each other. So what happens when two writes are genuinely concurrent, and "most recent" is just a lie your clocks are telling you?

That's Version Conflicts, and it's next — and it's where we finally go back for Ana's hat.