Version Conflicts

Ana adds a hat to her shopping cart.

Two seconds later — having seen the hat sitting there, having thought about it — she adds a lamp.

Here is her cart.

The lamp is gone, and Ana's cart now contains the item she added first. No error was raised. Nothing appeared in a log. The database did exactly what it is designed to do.

This is last-write-wins, and it is the default conflict-resolution strategy of an enormous amount of the software you use. And I want you to notice something about the sentence I just used to describe it, because it is a lie hiding in plain sight.

The last write did not win. The lamp was the last write. The lamp lost.

We have been circling this for four lessons. Replication Topologies (#21) showed you a conflict being born and promised we'd come back for it. Quorums: N, R, W (#23) said "what to do when two writes race is Version Conflicts." CAP, Properly (#25) watched a fully-acknowledged quorum write get destroyed by a lonely single-node write and told you flatly: a timestamp is not a vote. And Read Repair & Hinted Handoff (#27) caught the database picking a winner "based on the timestamps — the most recent replica is selected" and asked the question this lesson exists to answer:

Most recent according to whom?

Today we settle it. And at the end, we go back for Ana's hat.

Start with the thing nobody looks at. What is a write timestamp, actually?

In Cassandra you can just ask:

It's a wall clock reading. 1784046055456546 — microseconds since 1970, taken from a machine, at the moment of the write.

So when your database decides which of two writes survives and which one is destroyed forever, the thing it consults — the tiebreaker, the judge, the final authority — is a computer's opinion of what time it is.

And look at the precision of that opinion. Sixteen digits. It is quoting you a moment to the microsecond — one millionth of a second. Now ask yourself the question nobody asks: is it accurate to a microsecond?

It is not, and it is not close. A well-behaved machine synchronised by NTP over a LAN typically sits within a few milliseconds of true time — call it 1–10ms on a good day. That is already a thousand times coarser than the number it is printing. Over the public internet, tens of milliseconds. And on a machine whose NTP daemon has quietly died, there is no bound at all: a quartz crystal left to itself drifts steadily, in one direction, forever.

Those last six digits are not measurement. They are decoration. The database prints them with a straight face and then uses them to decide which of your customers' writes to destroy.

Google is the proof of how hard this is to fix. To get an actually trustworthy clock for Spanner, they put GPS receivers and atomic clocks in every datacenter — and even then their TrueTime API refuses to return a timestamp. It returns an interval: "now is somewhere between these two instants," with an uncertainty ε of roughly 1–7 milliseconds, averaging about 4ms. When Spanner needs a write to be genuinely ordered, it waits out that interval before acknowledging — it spends real latency, on every commit, buying the honesty that your WRITETIME column simply asserts.

You do not have atomic clocks. Which brings us to the demonstration.

One of Ana's requests lands on a node whose clock is five seconds fast.

That's not a contrived scenario and it's not a trick. It's a VM whose NTP daemon quietly died three weeks ago, and nobody noticed, because nothing in your stack alerts on clock drift. A node whose clock is five seconds fast doesn't do anything dramatic. It just stamps every single write five seconds into the future. Automatically. Forever. Until somebody fixes it.

Ana's lamp genuinely happened later. In the real world, in physics, in the only order that actually exists — she saw the hat, and then she added the lamp. The lamp is newer information. The lamp is what she wants.

The lamp is destroyed, because the hat's node had a faster clock.

And notice that this is not bad luck. It's arithmetic, and you can do it on your fingers. The hat was written 2 seconds earlier but on a clock running 5 seconds fast, so it carries a stamp 3 seconds ahead of the lamp's — exactly the 1784046079061211 versus 1784046076061211 you can see above, a gap of precisely 3,000,000 microseconds, pointing the wrong way.

Any skew larger than the real gap between two writes will invert their order. The writes are 2 seconds apart; the clocks are 5 seconds apart; 5 beats 2, so the order flips. And the real gap between two concurrent writes to the same key is usually milliseconds — which means it takes almost no drift at all to start losing the wrong ones.

"Last"-write-wins does not pick the last write. It picks the biggest number.

Sit with the consequence of that, because it's worse than a one-off bug. A node with a fast clock wins every race it ever enters. Every write it takes is stamped in the future, so it beats every write from every honest node in your fleet, on every key, for as long as the drift persists. It's not losing you a write. It is quietly establishing itself as the winner of all disagreements.

And there's a detail here I want you to notice, because it should be genuinely unsettling. In CAP, Properly (#25), Ana's hat was the victim — a fully-acknowledged quorum write, murdered by a lamp with a later timestamp. Here, the hat is the killer.

Which one dies depends entirely on whose clock happens to be fast. It's a coin flip, and NTP is tossing it.

Your instinct at this point is probably: fine, but surely the database knows this happened? Surely there's something I can look at — a warning, a counter, a flag, a second version of the row sitting somewhere?

I went and checked. All of it.

The lamp is not hiding somewhere. It was not archived, quarantined, or flagged. It is gone, and there is nothing anywhere in the system that records that it was ever there.

Cassandra has no vocabulary for "these two writes are concurrent." It has one slot, one number, and a greater-than sign.

It didn't resolve the conflict. It never knew there was one. From the database's point of view, nothing unusual happened at all: it received two writes to the same cell, and it kept the one with the larger timestamp, exactly as documented.

One quick honesty note, because it explains why so many people are convinced Cassandra handles this fine. Cassandra resolves conflicts per cell, not per row. So:

Two concurrent writes to different columns both survive, because each cell carries its own timestamp and is resolved independently. That's genuinely clever, and it means that in normal operation — where two users are usually editing different fields — nothing bad happens and everybody goes home happy.

Cassandra isn't handling your conflicts. It's usually just getting lucky. Touch the same field as somebody else, and one of you dies.

So why is the default this bad? Are database engineers stupid?

No — and the reason is a genuinely deep one, and Amazon wrote it down in the Dynamo paper in 2007. They ask: who should perform conflict resolution, the data store or the application? And then:

"If conflict resolution is done by the data store, its choices are rather limited. In such cases, the data store can only use simple policies, such as 'last write wins', to resolve conflicting updates. On the other hand, since the application is aware of the data schema it can decide on the conflict resolution method that is best suited for its client's experience. For instance, the application that maintains customer shopping carts can choose to 'merge' the conflicting versions and return a single unified shopping cart."

Read that again. The database's choices are rather limited. Not because it's badly written — because of what it is.

Your database is holding two byte-arrays and two numbers. It has no idea whether those bytes are a shopping cart (in which case: merge them, obviously, she wants both), or a bank balance (in which case: dear God do not merge them), or a display name (in which case: just take the newer one, it's fine). It cannot know, because only your application knows what your data means.

So it does the only thing it can do with two numbers and no semantics: it compares them, and throws one away.

Last-write-wins is not a strategy. It is what a database does when it doesn't know what your data means.

It isn't conflict resolution. It's conflict destruction, wearing the costume of a policy.

And Amazon is honest about why everybody ends up here anyway:

"some application developers may not want to write their own conflict resolution mechanisms and choose to push it down to the data store, which in turn chooses a simple policy such as 'last write wins'."

Which is to say: merging is work, and the default is free, and defaults win.

There's a deeper problem than skewed clocks, and it doesn't go away even if you fix NTP, and I want to make sure you see it — because if you only take away "clocks are unreliable" you'll go and buy better clocks and still lose data.

Last-write-wins asks: "which of these two writes is later?"

For two concurrent writes — two writes where neither node had any idea the other one was happening — that question does not have an answer. Not a hard-to-find answer. No answer. The two events happened in different places, with no information passing between them. Nothing in the universe orders them. Asking which came "first" is like asking which of two people, in two different cities, sneezed first — you can consult clocks, but you're inventing a fact, not discovering one.

The right question — the one Leslie Lamport taught us to ask — is completely different:

Did either of these two writes know about the other?

If write B was made by somebody who had seen write A, then B descends from A. B is a considered update. B should win, and A can be safely forgotten — Amazon calls that syntactic reconciliation, and it's the easy case, and it's most of them.

But if neither write saw the other, they are concurrent. They are two branches of history. And now:

"the system cannot reconcile the multiple versions of the same object and the client must perform the reconciliation in order to collapse multiple branches of data evolution back into one (semantic reconciliation)."

Those surviving branches have a name: siblings. A database that keeps them is telling you something honest and slightly rude — "two things happened, I have no idea which one you meant, here are both, you sort it out." Dynamo does this, and so does Riak, the open-source database built directly from the Dynamo paper: a read can hand your application back two values for one key and expect it to cope. Programmers tend to hate this the first time they meet it. It is, nonetheless, the only answer that is true, and you should notice that the database offering it is the only one in this lesson that has not destroyed anything.

A timestamp cannot distinguish these two cases. It gives you a number in both. It will happily tell you that a considered update and a blind collision are the same kind of event, and hand you the bigger number with total confidence.

There is a structure that can tell them apart — it tracks what each writer had seen, rather than what its clock said — and it's called a vector clock. Dynamo uses them precisely to answer "is this an ancestor, or is this a conflict?" We build them properly in Container 3. For today, what matters is knowing that the question exists, and that your timestamp is not asking it.

Right. Four lessons ago I destroyed this woman's hat, and I said we'd come back for it. Let's fix it — and the fix is not the one you're expecting.

It is not a better clock. It is not a stronger consistency level. It is not a different database.

The data loss was never the database's fault. It was the schema's fault.

Look again at what we asked Cassandra to store:

items text        -- 'ONE BOOK + A HAT'

One opaque string. A single slot. We took a thing that is fundamentally an accumulation — a shopping cart is built up, item by item, and every item matters — and we modelled it as a snapshot: one value that gets wholly replaced each time.

And once you've done that, two additions have no choice but to fight over one slot. We didn't just permit the data loss. We designed it in.

So model it as what it actually is. A set.

Nothing was lost.

The lower timestamp didn't lose — because there was nothing to lose to. These two writes were never in competition. items + {'HAT'} and items + {'LAMP'} are not two rival claims about what the cart is; they're two independent contributions to what the cart contains. Cassandra merges collections by union, and:

A union has no losers.

And here's the part I find genuinely beautiful. The clock is still wrong. That node is still five seconds fast. It is still stamping every write into the future. Nobody fixed NTP.

It just doesn't matter any more. We removed the fight, so there is nothing for the broken clock to decide.

That is the cheapest, most under-used fix in this entire field. Not a better clock — a data model in which the conflict cannot occur.

(And if it's nagging at you that "a set that merges by union" sounds suspiciously like it has a name — it does. That's a CRDT, and we build them properly in Container 3. You just met your first one.)

Before you go and turn everything into a set, let me collect the bill — because there is one, and Amazon states it in the Dynamo paper with admirable bluntness:

"Using this reconciliation mechanism, an 'add to cart' operation is never lost. However, deleted items can resurface."

So I went and tried to make it happen. Ana is on a train. Her connection is bad and she's looking at a slightly stale cart, so she re-adds the hat, thinking it didn't take. Then she changes her mind about the hat entirely and removes it — and that request lands on a node with an honest clock.

Her explicit, deliberate, later removal lost to a re-add with a bigger number. She removed the hat and the hat is in her cart.

You have seen this exact shape before. It is the zombie from Read Repair & Hinted Handoff (#27) — a delete that gets undone because the surviving evidence says "something", and something beats nothing. In #27 that resurrection was an accident of tombstone expiry. Here it is a design choice.

And it's a good design choice, which is the whole point. Amazon looked at these two failure modes and made a business decision:

• A lost item is a lost sale. The customer never notices; you just quietly earn less money.
• A resurrected item is a mildly annoyed customer who removes it again. Nobody dies, and you keep the sale on the other three things in the basket.

They didn't solve the conflict. They priced it — exactly like the ATM in CAP, Properly (#25) that hands you $200 during an outage and charges an overdraft fee later.

Every strategy loses something. You do not get to choose whether to lose. You only get to choose what.

Here's the thing to walk away with. When you're staring at a field in a schema and wondering whether last-write-wins is going to quietly eat your data, ask one question:

Is this value a SNAPSHOT, or an ACCUMULATION?

A SNAPSHOT is a value where the new one genuinely replaces the old one, and the old one is worth nothing. A user's display name. A device's latest temperature reading. A feature flag. A cached rendering. The current price of a product.

For snapshots, last-write-wins isn't merely acceptable — it's correct. Ana's display name says 'ANA' and she fixes it to 'ANNA'. Nobody wants both. Nobody wants a merge of both. There is no lost intent hiding in the old value, because the whole point of the write was to throw the old value away — so when the database destroys the loser, it destroys exactly the thing you asked it to destroy. This is the world Cassandra was built for, and it's why Cassandra is an excellent database. Run the lab below with two names instead of two cart items and watch last-write-wins do the right thing, effortlessly, for free.

AN ACCUMULATION is a value that is built up from contributions, where every contribution carries meaning. A shopping cart. A set of permissions. A counter. A comment thread. A list of tags. A user's saved addresses.

For accumulations, last-write-wins is data loss. Every concurrent contribution except one is destroyed, silently, with no error and no trace. And the default is last-write-wins.

If you would be upset to lose one of two concurrent writes, then last-write-wins is the wrong answer — and it is what you are currently using.

Two clients. One cart. You control the clocks.

Below you can set how far each node's clock has drifted, type what each client writes, and choose how the database resolves the collision. Everything else follows the rules exactly as a real database applies them.

Try this first. Leave both clocks correct and make the two writes genuinely sequential — one after the other. Last-write-wins does the right thing, and everything is lovely. Now push one node's clock five seconds fast and do the identical thing. Watch the write that happened second get destroyed by the write that happened first.

Then switch the strategy to keep both and see the conflict for what it actually is: not a puzzle the database solved, but a question it was never able to ask — two branches of history, sitting there, waiting for somebody who knows what a shopping cart means to reconcile them.

Then switch to union-merge and get Ana's hat back. And then — because I want you to feel the whole trade and not just the happy half — remove an item while the other client re-adds it, and watch it come back from the dead.

There is no setting on that panel that loses nothing. Go and look for it.

Go and find your accumulations. Open your schema and look for every field that is really a collection of contributions wearing the costume of a single value: a cart serialised into a JSON blob, a comma-separated permissions string, a tags column, a counter you read-modify-write. Every one of those is a place where two users can collide and one of them silently loses. They are not rare, and they are not exotic. They are in every codebase I have ever opened.

Fix the model before you fix the clock. A set, a list, a proper counter type, a child table — any of these turns a fight over one slot into an accumulation with no losers. It is almost always cheaper than the alternatives and it removes the failure mode instead of managing it.

Then, alert on clock drift. Genuinely — go and check whether you have any monitoring on NTP offset at all. Most teams have none, which means a node can sit five seconds ahead for a month and win every conflict in your cluster while every dashboard stays green. Google, with GPS receivers and atomic clocks in every datacenter, still refuses to claim it knows the time to better than a few milliseconds — Spanner's TrueTime returns an interval rather than a timestamp, and the system deliberately waits out the uncertainty before it will claim an ordering. If Google won't pretend, your EC2 instance definitely shouldn't be.

And when you do choose last-write-wins, choose it out loud. Write it down. "Concurrent edits to a display name resolve last-write-wins; we accept that one of them is discarded." That is a perfectly good engineering decision. What is not good is arriving there by accident, because it was the default, and then finding out during an incident that your shopping carts have been eating themselves for two years.

"Last-write-wins resolves conflicts." It destroys them. It doesn't resolve, detect, log or report anything — it compares two numbers and silently discards a write.

"The last write wins." Measured: the write that happened second lost. The biggest timestamp wins, and timestamps come from clocks that disagree. A node with a fast clock wins every race it ever enters, on every key, until somebody fixes NTP.

"Clock skew is a rounding error." Google — with atomic clocks — allows a few milliseconds of uncertainty and waits it out rather than pretend. A neglected VM can be seconds off. I used five, and nothing in the entire stack complained.

"My database handles conflicts." Cassandra cannot even represent one: no sibling, no error, no trace on disk. It resolves per cell, which is why it usually looks like it's working — most collisions happen on different fields. It isn't handling your conflicts; it's getting lucky.

"This is the database's fault." Mostly it is the schema's. A cart modelled as an opaque string forces two additions to fight over one slot. Modelled as a set, the conflict cannot occur — and the broken clock stops mattering.

"So union-merge is the safe option." It has its own bill, and Amazon prints it: "deleted items can resurface." Measured — Ana's removed hat came back. Every strategy loses something. You only get to choose what.

"Stronger consistency would have prevented this." Not necessarily. Two genuinely concurrent writes are concurrent regardless of your consistency level — a quorum tells you who saw the write, not which write was meant to come first. (A single leader would prevent it — by paying the coordination bill from CAP, Properly (#25) and PACELC (#26) instead.)

• A write timestamp is a wall-clock reading. Measured: WRITETIME is microseconds since the epoch, taken from a machine's opinion of what time it is.
• ⭐ "Last"-write-wins does not pick the last write. It picks the biggest number. Measured: with one node's clock 5 seconds fast, the write that happened second was destroyed by the write that happened first. A fast-clocked node wins every race it enters.
• The database cannot even see the conflict. No error, no sibling, no trace on disk. "It has one slot, one number, and a greater-than sign." It didn't resolve the conflict — it never knew there was one.
• ⭐ LWW is what a database does when it doesn't know what your data means. Dynamo: "If conflict resolution is done by the data store, its choices are rather limited… the data store can only use simple policies, such as 'last write wins'." Only your application knows whether those bytes are a cart (merge!) or a balance (don't).
• The real question isn't "which is later?" — it's "did either write know about the other?" If yes, one descends from the other and the newer wins (syntactic). If no, they are concurrent, there is no "later", and only your app can reconcile them (semantic). A timestamp cannot tell these apart. Vector clocks can — that's Container 3.
• ⭐⭐ THE FIX IS USUALLY THE SCHEMA. Ana's hat came back the moment we modelled the cart as a set<text> instead of one opaque string: {'BOOK', 'HAT', 'LAMP'}. A union has no losers — and the clock is still wrong, it just no longer matters.
• But every strategy has a bill. Union-merge means "deleted items can resurface" — measured: Ana's removed hat came back. Amazon chose that on purpose: a lost item is a lost sale.
• ⭐ THE TEST: is this value a SNAPSHOT or an ACCUMULATION? Snapshot (a display name, a sensor reading) → LWW is correct. Accumulation (a cart, permissions, a counter) → LWW is data loss — and it is the default.

That's §3. We have taken replication apart from every angle: who may accept a write, how long you wait, how many must agree, what "consistent" even means, what breaks when the network splits, what it costs when it doesn't, how the pieces heal, and — today — what happens when two of them genuinely disagree.

Now stop reading about it. The next lesson is a codelab: you're going to build a replicated Postgres, kill the leader, and promote the follower — with your own hands, on your own machine.