The Clock-Sync Problem

The checkpoint closed with a warning: all through the last stretch you read logs and told stories like “the standby promoted after the primary froze” as if every machine agreed on what after means. They don't. And you've already watched the body hit the floor: in Version Conflicts, a node whose clock ran five seconds fast stamped an older write as newer, and the database kept the wrong one, politely, exactly as documented. A node with a fast clock wins every race it ever enters.

That lesson taught you what the damage looks like. This one goes after the weapon. Where does a machine's idea of “now” even come from? Why does it slip? Why doesn't syncing fix it — not “fix it badly”, but cannot fix it? And what does time look like when a system finally stops lying to itself about it?

By the end you'll know why the best-funded engineering organizations on earth, with GPS receivers and atomic clocks in their datacenters, refuse to answer the question “what time is it?” with a number.

Inside almost every server is a sliver of quartz cut like a tuning fork, vibrating 32,768 times a second. Count the vibrations, and you have a clock. The catch is in the word almost: no two slivers vibrate at exactly the same rate. The error is measured in parts per million, and the arithmetic is unforgiving:

• 1 ppm wrong ≈ 86 milliseconds lost or gained per day.
• A typical cheap crystal is off by 10–50 ppm: up to about 4.3 seconds a day.
• Temperature is the biggest dial: a crystal's rate bends with heat, so the machine in the hot rack and the machine by the cold aisle don't just disagree — they disagree by an amount that changes with the weather inside your datacenter.

So a fleet of servers is a room full of metronomes, each keeping its own slightly wrong beat, each wrong by a different amount in a different direction, wandering as the room warms and cools. Left alone for a week, they'd be seconds apart. That's drift, and it's not a defect you can file a ticket about. It's what quartz is.

Two familiar characters get new faces here. The zombie leader from Network Partitions & Split Brain woke from its pause convinced that no time has passed — that conviction was a frozen clock, resuming. And the murder weapon in Version Conflicts was a VM whose time daemon had quietly died three weeks earlier: drift with nobody watching it, because, as that lesson put it, nothing in your stack alerts on clock drift. Hold that thought; it becomes a to-do at the end of this lesson.

The obvious answer to drift is to ask a better clock. That's the Network Time Protocol, and it's genuinely good engineering, as long as you understand what it actually does.

NTP arranges the world's clocks in a pyramid of strata: at the top, atomic clocks and GPS receivers; below them, servers that sync to those; below them, servers that sync to those. Your machine picks servers to ask, and asks like this: it sends a request stamped with its own send time, and gets back the server's receive time, the server's reply time, and stamps its own arrival time: four timestamps for one round trip. From those four numbers it computes how far its clock is off.

Look at what that computation has to assume, though. Your machine knows the total round trip took, say, ten milliseconds, but not how the ten was split. The math charges half the trip to each direction. If the path really is symmetric, the answer is exact. If the request crawled through a congested uplink in nine milliseconds and the reply sprinted back in one, the estimate lands four milliseconds wrong — and nothing on your machine can detect it, because the error lives precisely in the assumption that made the math possible. The figure below is the whole trap in one picture.

What the daemon does with the estimate has its own personality:

• Small correction (under about 128 ms): it slews — bends the tick rate slightly, up to 500 ppm, so the clock glides to the right value without ever jumping.
• Big correction: it steps — the clock simply jumps to the new value. And if the clock was ahead, it jumps backwards: your machine reads 12:00:01, and a moment later it is 11:59:59 again. The same second happens twice.
• Huge correction (around 1,000 seconds or more): the daemon refuses entirely and gives up, on the theory that something is wrong enough that a human should look.

With healthy peers, NTP holds a machine within a millisecond or two on a quiet LAN, and within one to ten milliseconds across the internet, with a tail that stretches past a hundred under congestion. That is real skill. It is also nowhere near “agreement”: your writes are microseconds apart, your clocks are milliseconds apart, and each machine's belief about its own accuracy is itself an estimate resting on an assumption it can't check. Syncing brings clocks closer. It cannot make them the same, and — this is the part worth tattooing somewhere — a freshly synced machine that is wrong has no way to know it.

Everything above describes the wall clock — the one that tries to match the outside world, the one NTP disciplines, the one that can be slewed, stepped, and yanked backwards. Your operating system quietly offers a second clock, and knowing which one to reach for is one of those small distinctions that separates code that survives from code that makes the postmortem: the monotonic clock starts counting at boot and only ever counts up. It drifts like any quartz, but it never jumps and never repeats. Its readings mean nothing to any other machine — and nothing to this machine after a reboot — but the difference between two of its readings is a trustworthy duration.

The rule: durations on the monotonic clock; moments in time on the wall clock, held with suspicion. Every timeout, every latency measurement, every “how long since the last heartbeat?” belongs to the monotonic clock, because the wall clock's answer to “how long?” can be negative.

That is not a theoretical jab. On New Year's Day 2017, a leap second nudged wall clocks around the world, and inside Cloudflare's DNS software — written in Go, whose standard time library at that point followed the wall clock — “a number went negative when it should always have been, at worst, zero,” as their postmortem put it. That negative duration was fed into a random-number call that panics on negative input. At the peak, about 0.2% of DNS queries to one of the internet's biggest DNS providers were failing. The repair took ninety minutes to reach the worst machines. The fix was one character.

The epilogue says even more: two releases later, Go changed its standard library so that every time value silently carries both clocks: wall for meaning, monotonic for arithmetic. The language itself now refuses to measure a duration with the clock that lies.

It gets one level worse. The reference at the top of the pyramid — UTC itself — occasionally steps on purpose. The earth's rotation is slightly irregular, so every few years the timekeepers of the world insert a leap second: somewhere a minute contains 61 seconds, and every clock on the planet is asked to cope.

On June 30, 2012, coping went badly. A bug in the Linux kernel's timekeeping meant the inserted second could leave locks in a state where processes spun forever: machines alive, fans roaring, doing nothing. Mozilla, Reddit, LinkedIn, Gawker, and Yelp all took hits that night. The Amadeus Altéa reservation system went down for about an hour, and staff at Qantas and Virgin Australia checked passengers in by hand while planes waited. One second, inserted with years of advance notice, on a published schedule — and airlines fell back to paper.

The fix that won the argument is charming: don't insert the second — smear it. Google's time servers, and now Amazon's, stretch the extra second across the 24 hours around the leap, noon to noon, running every clock imperceptibly slow. How imperceptibly? The smear bends clock frequency by about 11.6 ppm — smaller than the manufacturing error of the quartz in your servers. The correction hides inside the noise the clocks already have. It's an honest trade dressed as a trick: during the smear, your whole fleet is deliberately wrong about UTC by up to half a second, but it's wrong together, smoothly, with no step and no 61-second minute. Which tells you what the industry actually decided matters: not being right, but never jumping.

Time to hold two clocks in your own hands. The lab below runs two machines at sixty times real speed, a minute of machine time for every second you watch, because honest drift is measured in microseconds per second and you'd be here all week.

Try this first: give machine A a hot rack and machine B a good crystal, sync them both, and just watch — the true-skew meter climbs as the two metronomes beat apart, and each machine's ±ε band (its honest “I could be this far off”) widens as its last sync ages. Sync again and both collapse. That sawtooth — confidence decaying, snapping back, decaying — is the actual shape of time on every machine you own.

Then run the experiment this lesson is named for. Turn on the congested uplink and hit sync: the exchange completes, machine A reports itself synced and healthy, and the god-view meter (a privilege only a lab has) shows the offset that path asymmetry just smuggled into the estimate. In production there is no god view. That gap between what the machine believes and what is true is the whole reason this lesson exists.

Finally, ask the question that decides designs: type how far apart two writes might land (ten milliseconds? one?) and watch the verdict: orderable by these clocks, or not, as ε grows and shrinks. Switch machine A to the dead time daemon (the murder weapon from Version Conflicts, resurrected as a preset), give it a few seconds to wander past the step threshold, then sync it to see a step: the wall clock jumps backwards and reads the same moment twice, while the monotonic counter beneath it climbs straight through, untouched. That one animation is the two-clocks rule made visible.

Everything so far says a timestamp is a point-shaped claim about a fuzzy fact. The most honest engineering answer is also the simplest to say: stop returning points.

Google's Spanner database is the famous version. Its TrueTime service, asked what time it is, returns an interval — earliest and latest — meaning: the true time is in here, guaranteed. Behind it stands serious hardware: every datacenter runs time masters fed by GPS receivers, plus a minority armed with atomic clocks (the paper genuinely calls them Armageddon masters). Every machine polls a spread of masters every 30 seconds, discards the liars, and budgets for its own drift in between. The interval's width, called ε, sawtooths between roughly one and seven milliseconds, averaging about four: tight just after a poll, widening as the crystal wanders, snapping tight again.

And here is the move that makes it a database primitive rather than a nicer clock: when Spanner commits a transaction, it assigns the timestamp and then waits out the uncertainty — commit-wait — refusing to acknowledge until the interval containing “now” has entirely passed the assigned time. A few milliseconds of deliberate patience buys a guarantee most systems can't make at any price: if transaction B starts after transaction A finishes, anywhere on earth, B's timestamp is genuinely larger. Google did not fix the clocks. It priced the doubt and paid it, on every single commit.

You can rent the same honesty without the atomic clocks: AWS's time service comes with a library that hands your code a bounded interval instead of a bare timestamp, and Meta has pushed hardware-timestamped sync into the microsecond range across its fleet. The gear keeps improving. The shape of the answer doesn't: an honest clock tells you how wrong it might be, and a system that wants correctness waits out the doubt rather than pretending it away.

But notice the ceiling, because Version Conflicts already pressed your face against it: when two writers act independently (neither having seen the other), no interval, however tight, makes “who was first” a real question. You'd be inventing a fact, not discovering one. Clocks, even honest ones, answer when. The question your data actually needed answered was what had each writer seen. Hold that.

• Every clock is a slightly wrong metronome. Quartz drifts tens of parts per million and bends with temperature: seconds per day, unsynced, in each machine's own direction.
• Syncing is estimating, not agreeing. NTP charges half the round trip to each leg; path asymmetry hides inside that assumption as error your machine cannot detect. Closer, never equal — and a wrong machine that just synced believes it's right.
• Two clocks, one rule: durations on monotonic, moments on wall — with suspicion. The wall clock slews, steps, and runs backwards; Cloudflare lost 0.2% of global DNS queries to one negative duration, and the fix was one character.
• Even UTC steps. The 2012 leap second livelocked kernels and put airline check-in on paper; the industry's answer — smear it at 11.6 ppm, below the quartz noise floor — chose “never jump” over “be right”.
• Honest time is an interval. TrueTime returns [earliest, latest] and Spanner waits out ε on every commit; AWS rents you the same shape. The honesty, not the hardware, is the fix.
• The discipline this buys you: measure timeouts on the monotonic clock, keep your fleet synced, and — paying a debt from Version Conflicts — alert on clock drift like you alert on disk space, because the murder weapon there was a time daemon that died with nobody watching.

One more thing. Every fix in this lesson made the clock better (tighter, honester, smoother), and every one of them still ran into the same wall: some ordering questions aren't about time at all. “Did the reply come after the request?” doesn't need a wall clock; it needs the arrow between two events. The next lesson makes that idea precise, and it's the pivot the rest of this stretch turns on: stop asking when — ask before or after. That's Physical vs Logical Time.