Read Replicas & Replication Lag

Introduction

In the last lesson you did the right thing. Your reads were saturating one machine, so — after checking it was a busy problem and not a slow one — you added a read replica. Your read throughput doubled. You went home.

Then this ticket came in:

"I changed my display name to 'Ana', hit save, it said saved, I refreshed the page — and it's back to 'Anastasia'. Your app lost my change."

You check the database. The name is 'Ana'. It saved perfectly. You refresh — you see 'Ana'. You cannot reproduce it. You close the ticket as cannot reproduce.

It will keep happening, to different users, forever, and you will never catch it in your own testing — because the bug is not in the database, and it is not in your code. It is in a decision your infrastructure is making thousands of times a second, silently: for this read, from this user, right now — which machine answers it?

That decision is the whole lesson.

The vanishing write, drawn as two steps. Step one, SAVE: a user sends the new value 'Ana' to the LEADER database, which stores display_name = 'Ana' and confirms it saved. But 272 bytes of write-ahead log are still in flight to the replica. Step two, RELOAD: the very next read is routed to the REPLICA, which has not replayed that write yet, so the user sees the OLD value 'Anastasia' — their own change is gone. A banner across the foot states the measured reality: the replica is 272 bytes behind, and that is on an idle system; under load it hits 34 megabytes behind, because replication lag spikes exactly when you are busy.

The Write That Vanished

Here is exactly what happened to Ana, on a real Postgres, in slow motion.

She hits save. The write goes where all writes must go — the leader. The leader stores it and confirms. So far, flawless.

-- on the LEADER, immediately after she saves:
SELECT display_name FROM profile WHERE user_id = 42;
  ->  Ana        ✅ saved, no error, everyone's happy

Now her browser reloads. That reload is a read, and your infrastructure — doing exactly what you told it to do to scale your reads — sends it to a replica. But the write hasn't arrived at that replica yet. It's a few hundred bytes of write-ahead log, still in flight.

-- her reload, routed to the REPLICA, a heartbeat later:
SELECT display_name FROM profile WHERE user_id = 42;
  ->  Anastasia  ⚠️ her own write is GONE

She saved 'Ana', reloaded, and watched her change disappear. This is a read-your-writes violation, and it is the single most common bug introduced by read replicas.

In The Consistency Spectrum (#24) you met this as an anomaly — a theoretical thing a weak consistency model permits. Here it is as what it actually is: a support ticket, from a real person, who watched your software eat their work. The theory just grew a face.

The Number That Is the Bug

How far behind was that replica? Postgres will tell you, from the leader:

SELECT application_name, state,
       pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS bytes_behind,
       replay_lag
FROM pg_stat_replication;

 application_name | walreceiver
 state            | streaming
 bytes_behind     | 272                     <-- the whole bug, in one number
 replay_lag       | 00:00:00.027867         (27.9 ms)

272 bytes. 27.9 milliseconds. That is the size of the gap Ana fell into. Her reload arrived during a 27-millisecond window in which the replica had not yet caught up, and it lost.

And here is the part that makes this genuinely hard, the part that separates people who understand read replicas from people who have merely added one:

Lag Is Not a Number. It's a Number That Spikes.

Twenty-seven milliseconds sounds harmless. You might reasonably think: fine, there's a tiny window right after a write, most reads will miss it, I'll accept the occasional lost update.

That reasoning is a trap, because lag is not constant. It sits near zero when the system is quiet — and it explodes the moment the leader gets busy, because the replica has to replay every write the leader takes, and when the leader is taking writes faster than the replica can apply them, the replica falls further and further behind.

I measured it. Same leader, same replica. First idle, then during a burst of writes:

IDLE:                    0 bytes behind  ·  replay_lag  1.07 ms

DURING a write burst on the leader (sampled every 0.5s):
   t+0.5s    23 MB behind   ·  205 ms
   t+1.0s    16 MB behind   ·   84 ms
   t+1.5s    34 MB behind   ·  252 ms      <-- 0 bytes  ->  34 MEGABYTES
   t+2.0s     8 MB behind   ·  216 ms

The exact same replica went from 0 bytes behind to 34 megabytes behind — and it did so precisely when the system was busiest.

Now connect the two facts. Lag spikes when the system is busy. And when is the system busy? When the most users are on it — which means when the most users are writing, and then reading their own writes. The staleness window doesn't just exist; it widens at the exact moment the maximum number of people are exposed to it. This is why the bug feels random and rare in testing and is neither in production.

You cannot design around "lag is usually a few milliseconds." You have to design around its spikes.

The Fix Is Not a Faster Replica. It's Where You Send the Read.

The instinct is to make the lag smaller — a beefier replica, a faster network, synchronous_commit. Those help at the margin, and none of them fix the bug, because the bug is not that the replica is slow. The bug is that you sent Ana's read to a machine that couldn't answer it correctly.

So the real fix lives one layer up, in the thing that decided where her read went: the router. And the crucial realization — the one that reframes this whole problem — is that the routing decision is made per read, not once for the whole system:

The same replica is the right place to send one read and the wrong place to send the next one.

A read of last month's sales report? Send it to any replica; a few seconds of staleness is invisible. A read of the profile Ana edited four milliseconds ago? That one, and only that one, needs special handling. The router is the brain of a read-scaled system, and its entire job is to ask, for every read: can this one tolerate stale data?

There are three standard answers, and — this being systems design — each is correct for a different situation and each has a bill.

The routing decision, drawn as a router at the top branching into three read-routing policies, each with what it buys you and its bill. The first, in red, is read from the leader: you get an always-correct answer, but the read never touches a replica, so you have deleted your read scaling — a scalpel, not a blanket policy. The second, in green, is wait-for-LSN: you get a correct answer served from a replica; the bill is that the read waits for the replica to catch up, and this is what AWS Aurora and Vitess actually ship — ship this. The third, in amber, is a sticky session: it fixes monotonic reads, a different bug, not read-your-writes, and it breaks on failover — a scalpel. The header states the point: the fix is never a faster replica, it is where you send the read, and every routing choice is correct for one bug and has a bill.

The Three Routing Fixes, and What Each One Costs

Fix 1 · Read from the leader. For reads that must reflect the user's own recent write, route them back to the leader. Ana's profile read goes to the leader; it sees 'Ana'; the bug is gone.

The bill: that read never touches a replica. You just took the read-scaling you bought in #30 and gave it back for this read. As a scalpel — "send a user's read to the leader for a few seconds after they write" — it's perfect. As a blanket policy it is self-defeating: you paid for replicas and then routed around them.

Fix 2 · Wait-for-LSN — the production answer. This is the clever one, and it's what the grown-up systems actually ship. When Ana's write commits, the leader hands back the write's position in the log — its LSN. Her session remembers it. When her next read arrives at a replica, the router first checks: has this replica replayed past Ana's LSN yet? If not, it waits for it to catch up, then reads.

-- the write returns its commit position:
SELECT pg_current_wal_lsn();          ->  0/3000110

-- the read first checks the replica has caught up to it:
SELECT pg_last_wal_replay_lsn() >= '0/3000110'::pg_lsn;
  ->  f     -- not yet: WAIT...
  ->  t     -- now serve the read from the replica.  ✅ correct, AND still on a replica

The bill: this one read waits — bounded by exactly how far behind the replica is (which, as you just saw, spikes under load). But it stays on a replica, so your read-scaling survives. This is the mechanism behind AWS Aurora's read-after-write consistency and Vitess/ProxySQL's GTID tracking. It is the answer you should reach for by default.

Fix 3 · Sticky sessions — for a different bug. Pin each user to the same replica (usually by hashing their user id). This fixes monotonic reads: the anomaly where a user reads a fresh replica, then a stale one, and watches time run backwards — a comment appears, then vanishes, then reappears.

The bill: it does not give you read-your-writes — a user pinned to a replica that still lacks their write cannot see it. Different bug, different fix. And it breaks on failover: when that replica dies, the user is re-pinned to another one at a different position, and time can jump again.

Notice the shape of all three: there is no routing policy that is free, and there is no routing policy that is right for every read. Which is exactly why the decision has to be made per-read, by something that knows what each read is.

Route It Yourself

Below is Ana's exact situation, live. Type a new name, hit Save, then Reload — and watch the router physically send the read to the leader or a replica.

On the naive default, reload right after saving and watch your own change vanish, exactly as Ana did. Then hit the traffic spike and watch the replica fall from a few bytes to tens of megabytes behind — the staleness window widening in real time. Then try each routing policy and feel its bill: read-from-leader fixes it but the read abandons the replica; wait-for-LSN fixes it but the read has to wait out the lag; sticky fixes a different bug than the one you have.

The one question the router is really asking, on every single read, is the one you should be asking too: can this read tolerate stale data?

Rename yourself, hit Save, then Reload — and watch where the router sends the read. On the naive default (round-robin to a replica) your own new name vanishes, because the read lands on a replica that hasn't replayed your write. Hit the traffic-spike button and watch the lag balloon from a few bytes to tens of megabytes — exactly what happens on a real system when it gets busy. Then switch the routing policy: read-from-leader is correct but un-scales your reads; wait-for-LSN keeps the read on a replica but makes it wait; sticky sessions fix a different bug entirely. Every number is from a real Postgres.

The Three Guarantees You Are Actually Choosing Between

Step back and name what you're doing. When you scale reads with replicas, you are choosing, per read, which of three guarantees you need — and each maps to a specific bug and a specific fix:

the guarantee	the bug when it's missing	the fix
Read-your-writes	"it didn't save" (Ana)	read-from-leader / wait-for-LSN
Monotonic reads	"it keeps flickering" (time runs backward)	sticky sessions
Consistent-prefix	"the reply shows before the question"	keep related writes on one shard

That third one, consistent-prefix, is what breaks once your data is spread across multiple machines that each lag independently — and fixing it is the whole reason writes that must be ordered together have to live together. That's a thread we pick up when we start splitting data by key, in Container 3.

For now, the discipline is this: staleness is not a bug you fix once. It is a property you route around, one read at a time.

Mental-Model Corrections

"I added a replica, my reads are scaled, I'm done." You also inherited staleness — and pointed it straight at your users. Every replica is a new place a user's own write can fail to appear.

"Lag is usually a few milliseconds, so it's basically fine." Lag spikes under load (measured: 0 → 34 MB), and load correlates with writes, so the window is widest exactly when the most users are reading their own writes. Design for the spike, not the median.

"Just read everything from the leader to be safe." Then you have no read replicas. You paid for them and routed around them. Read-from-leader is a scalpel — this user, this read, for a few seconds — not a blanket.

"Synchronous replication removes the lag." It removes the loss window on failover (§3), not the read-path lag — and it makes your leader wait for the replica on every commit, which is its own tax (§3 measured the leader hanging when a synchronous replica died).

"Sticky sessions give me read-your-writes." They give you monotonic reads — a different bug. A user pinned to a replica that hasn't got their write still can't see it. And sticky breaks on failover.

"The replica is behind — that's the replica's fault." The replica is doing its job: replaying a log that is arriving faster than it can apply it, because the leader is busy. The fault, and the fix, live in the routing.

Key Takeaways

⭐⭐⭐ A read replica scales your reads and hands you staleness, pointed at your users. The instant a user reads their own recent write off a lagging replica, their change vanishes — measured: they saved 'Ana', reloaded, saw 'Anastasia', with the replica 272 bytes / 27.9 ms behind.
⭐⭐⭐ Lag is not a constant — it SPIKES under load. Measured: 0 bytes idle → 34 MB during a write burst. And load correlates with writes, so the staleness window is widest exactly when the most users are reading their own writes. Design for the spike.
⭐⭐ The fix is never a faster replica. It is a per-read ROUTING DECISION. The same replica is the right place for one read and the wrong place for the next. The router is the brain: for every read, it asks can this one tolerate stale data?
⭐ Three fixes, three bills: read-from-leader (correct, but un-scales that read — a scalpel) · wait-for-LSN (correct AND stays on a replica, but the read waits — what Aurora & Vitess ship, your default) · sticky sessions (fixes monotonic reads, a different bug; breaks on failover).
Name the guarantee you need per read: read-your-writes ("it didn't save") · monotonic reads ("it flickers") · consistent-prefix ("reply before question" — needs related writes on one shard, → C3).
Staleness is not a bug you fix once. It is a property you route around, one read at a time.

Next: Denormalization. You've been scaling reads by copying the database. Next you scale them by reshaping the data itself — pre-joining the answer so the read barely has to work. And you'll meet a new bill: the copies now live inside your own schema, and keeping them agreeing is your job.