Physical vs Logical Time

The previous lesson ended with its hands against a wall. Clocks, even honest ones, answer when, and the question your data actually needed answered was what had each writer seen. You watched engineering at its most heroic (atomic clocks, GPS antennas, deliberate waiting) buy tighter and tighter answers to a question that, for the cases that hurt, was never the real question.

This lesson makes the pivot the whole stretch turns on: stop asking when — ask before or after. It sounds like a smaller question. It is actually a different kind of question, with a different kind of answer: one that doesn't degrade with distance, doesn't drift with temperature, and doesn't need a satellite. By the end you'll have the idea that the rest of these lessons are built from — and you'll have watched a courtroom where a wall clock and a message graph rule on the same evidence and disagree.

Pull up the ordering questions that actually matter in a production system and look at their shape:

• Did the reply come after the request it answers?
• Did the customer add the item before checking out?
• Did the cancellation arrive after the parcel was already dispatched?
• Did this write know about that write — or were they strangers?

Not one of these needs a date. Nobody asks at what o'clock the reply happened; they ask whether it followed its request. Wall-clock time is how we've been approximating those questions — stamp everything, sort by stamp, hope. Within one machine the approximation is decent. Across machines you know exactly how it fails, because the previous lesson priced it: clocks agree to milliseconds on a good day, your events land microseconds apart, and the sort inverts. An engineer reading a merged log sees a service respond to a request that, according to the timestamps, hasn't arrived yet. Effect before cause, in writing.

So take the hint the questions have been giving all along. They are questions about order: about whether one event could have influenced another. And order, it turns out, has a source that never touched a quartz crystal.

In 1978, Leslie Lamport published Time, Clocks, and the Ordering of Events in a Distributed System — one of the most-cited papers in computer science, and its core move is small enough to write on an index card. Define happened-before (written a → b) using no clocks at all, only three rules:

1. Same machine: if a and b happen on the same machine and a comes first, then a → b. A machine may not know the time, but it certainly knows the order of its own actions.
2. Send before receive: if a is the sending of a message and b is its arrival, then a → b. A message cannot arrive before it was sent — not an assumption, a fact of causality.
3. Chains: if a → b and b → c, then a → c. Follow the arrows.

That's the whole definition, and notice what it's made of: program order and messages. Nothing else. Which encodes a claim deep enough to build the next several lessons on: in a distributed system, influence travels only through communication. A machine knows its own past, and it knows exactly what the messages it has received have told it. There is no other channel. If there is a chain of work and messages from a to b, then a could have shaped b — a is honestly, physically before b, in the only sense that affects your data. If there is no chain, no amount of clock precision will manufacture one.

The figure below is the three rules drawn once each; the third rule is where the power lives: chains cross the whole system, machine to machine, hop by hop, carrying before-ness with them.

Here is where the reframe earns its keep. Happened-before is a partial order: some pairs of events have a chain in neither direction. Lamport gave those pairs a name that trips everyone up on first contact: concurrent.

Concurrent does not mean simultaneous. Two concurrent events might be hours apart on the wall. It means something stronger and stranger: neither could have known about the other. No chain from a to b, no chain from b to a — two histories that had not yet met. Version Conflicts handed you this exact situation and the right instinct for it: asking which of two independent writers went first is like asking which of two people, in two different cities, sneezed first — you can consult clocks, but you're inventing a fact, not discovering one. Now you have the precise language for why. For concurrent events, “which came first” isn't merely unmeasured. It has no answer. There is no fact of the matter waiting behind better instruments.

If that idea feels like it rhymes with physics, it should. Lamport has said the paper came from exactly that place: “Special relativity teaches us that there is no invariant total ordering of events in space-time; different observers can disagree about which of two events happened first.” Your fleet lives in that universe. Messages are how light cones touch.

And this single idea redraws the map from the last several lessons. The split-brain cluster from Network Partitions & Split Brain? Two halves generating concurrent histories — that's why healing can't merge them. The doomed “last” in last-write-wins? A total order forced onto concurrent writes by lying instruments. The whole crime scene, one cause.

Time to watch the two definitions of “before” disagree in public.

The court below runs three machines with rigged clocks: A runs 80 milliseconds fast, C runs 120 milliseconds slow, which the previous lesson taught you is a perfectly ordinary Tuesday. Do some work on each machine, send a few messages, then click any two events. Two judges rule on the same pair:

Try this first: hit load the ambush, then click the send on machine A and its arrival on machine C. The wall-clock judge reads the stamps and rules that the arrival came first — the effect before its cause, in writing, exactly the inversion your merged logs produce. The causality judge walks the graph and lights up the chain, hop by hop, proving the opposite. One of these judges is describing your system. The other is describing two skewed pieces of quartz.

Then build the quieter and more important case yourself: two events on two machines with no message path between them, in either direction. The wall judge still answers — it always answers — but the causality judge tells you the truth: concurrent. Neither could have known. That refusal is not a weakness. It's the first honest answer in this whole stretch, and the next two lessons are about carrying it efficiently.

Name the two tools, because from here on the course will reach for both, deliberately:

Physical time is clock time — quartz, NTP, TrueTime intervals. It measures durations and speaks human. It is the right tool every time the question has seconds in it: timeouts and heartbeats (on the monotonic clock, as the previous lesson insisted), cache TTLs, “this lease is valid for ten seconds”, rate limits, “show me yesterday's orders”. And, paid for honestly, it can even buy ordering: commit-wait works precisely by waiting out the doubt until a real before-and-after exists. Physical time is not the villain of this stretch. It's a fine tool that was handed the wrong job.

Logical time is order time — happened-before made portable. No seconds, no durations, no meaning to a human reading a dashboard. It answers exactly one question — could this have influenced that? — and it answers it correctly, because it's built from the only thing order is actually made of out here: communication.

The rule of thumb the rest of the course will lean on: if the question contains seconds, reach for physical time; if the question contains “before”, reach for logical time. Most production incidents involving time trace back to someone using the first tool on the second kind of question.

• The reframe: stop asking when — ask before or after. Every ordering question that matters in production is a before-or-after wearing a timestamp costume.
• Happened-before needs no clocks. Three rules — same machine, send before receive, chains — and influence travels only through communication.
• It's a partial order, and the gaps have a name. Concurrent means causally independent: neither could have known about the other. Not “at the same time” — hours apart on the wall, possibly.
• For concurrent events, “which was first” has no answer. Not unmeasured — nonexistent. No clock precision manufactures a fact that isn't there; only a message can.
• Two tools, two jobs. Seconds in the question → physical time (durations, TTLs, leases, human time, commit-wait). “Before” in the question → logical time. The expensive incidents come from crossing the streams.

One thing is missing, and you can feel it: happened-before, as defined, is a relation: to check a → b you'd have to walk the whole message graph, every time, for every pair. Nobody ships a graph traversal in the write path. What you want is a little number each machine can carry, stamp on every event, and compare in one instruction — a number that encodes the arrows. That number exists, it's nine lines of code, and it's next: Lamport Timestamps.