Why Caching Wins

A single listing goes viral on your marketplace — a limited sneaker drop. Fifty thousand people are staring at the same product page: the title, the price, the seller's rating, "12 left." Every one of those page loads asks your database for the same rows, tens of thousands of times a minute. And your database — the single most expensive, hardest-to-scale box you own — dutifully runs the identical joins over and over, sweating harder with every viewer, inching toward falling over at the exact moment you're finally famous.

Here's the thing: that data barely changes. The price is the price; "12 left" is fine if it's a few seconds stale. So you do something almost embarrassingly simple — you keep the rendered listing in memory for thirty seconds. Now the first viewer pays the ~50-millisecond trip to the database, and the next fifty thousand get the answer in about a millisecond, straight from RAM. Fifty thousand database queries just became one. The page got roughly fifty times faster, and the database went from drowning to bored — from a single idea: remember the answer.

That's caching, and this section is about doing it well. But before the how, the why — because caching isn't free magic, and knowing exactly why and when it wins is what separates sprinkling caches everywhere (and creating a mess of stale data) from placing one where it turns a system around.

In this lesson: the latency ladder that makes caching worth it, the one property of real traffic that makes it actually work, the simple math of the payoff, and — just as important — the workloads where a cache buys you nothing.

Scope: who fills the cache and who owns the truth is the next lesson, Caching Patterns; what to throw out when it's full is Eviction; and keeping copies from going stale — the hard one — is Cache Invalidation. Here we're answering only: why cache at all?

You met the memory hierarchy back in the OS foundations — the ladder where every step closer to the CPU is wildly faster than the one below it. It's worth pinning up again, because caching is nothing more than this ladder, exploited on purpose:

• L1 CPU cache — about 1 nanosecond
• RAM — about 100 nanoseconds
• SSD random read — about 150 microseconds (~1,500× slower than RAM)
• Datacenter round trip — about 500 microseconds
• Disk seek — about 10 milliseconds (~100,000× slower than RAM)
• Cross-continent packet — about 150 milliseconds (~1,500,000× slower than RAM)

Those exponents are hard to feel, so borrow the classic trick: pretend one nanosecond were one second. Then an L1 read is a single heartbeat. A RAM read is about a minute and a half. An SSD read is almost two days. A disk seek is four months. And a packet to another continent and back is nearly five years. Reading from memory instead of disk isn't "a bit faster" — on this scale it's a minute-and-a-half versus a third of a year.

Now the move. Your source of truth — a database, over the network, backed by disk — lives near the bottom of that ladder. Caching keeps a copy of the answer near the top — in RAM, or on a machine right next to the user. You're not making the database faster; you're avoiding the trip to it for the reads that repeat. (That's the mirror image of the last section: async moved slow work off the request path; caching skips the slow work by remembering what it produced.)

A cache is small and your data is large — the whole listing catalog will never fit in RAM. So why does keeping a tiny slice of it in memory help at all? Because real traffic is nowhere near evenly spread. A small number of things are wanted a huge amount, and everything else is wanted rarely. The trending sneaker gets fifty thousand views; some obscure listing from 2019 gets one a week. This skew has a name — a Zipf distribution, the 80/20 rule in the wild — and it's the single property that makes caching work. Cache engineers call the busy keys hot keys, and there are always far fewer of them than you'd guess.

Here's how dramatic the effect is. Take a cache that holds just 100 keys out of a 10,000-key space — one percent of your data. If access is realistically skewed (Zipf), that one-percent cache serves about 78% of all requests. If access were perfectly uniform — every key equally likely — the identical cache would serve about 1%. Same cache, same size; the only difference is whether the traffic has locality. That 78%-versus-1% gap is the entire reason caching wins — and it's also the precondition. No skew, no hot keys, no win. A cache is a bet that the near future looks like the recent past, and for almost all real systems, it does.

Numbers on a page are one thing; watching the database load melt away is another. Here's a live read path — clients on the left, a cache in the middle, the database on the right. Every request that hits the cache bounces straight back from memory; every miss trudges on to the slow database and then fills the cache on its way home.

Do this first: leave the access pattern on HOT and drag the cache size up from almost nothing — watch the hit rate shoot up fast and the effective latency collapse while a small cache captures the hot keys. Then flip the pattern to UNIFORM and watch the whole thing die: the same cache now catches almost nothing, the latency snaps back to database speed, and the database is drowning again. That flip is the lesson — caching's entire payoff rides on locality. Keep an eye on the effective-latency number as the hit rate climbs past 90%: the last few percent buy you the most.

Put a number on it. The average latency of a cached read is just a weighted average of the two paths:

effective latency = hit_rate × (cache time) + miss_rate × (database time)

Say a cache read is ~1 ms and a database read is ~50 ms. At a 90% hit rate: 0.9 × 1 + 0.1 × 50 = 5.9 ms. Push it to 99%: 0.99 × 1 + 0.01 × 50 = 1.49 ms. Nine extra points of hit rate made it nearly four times faster — because the database is 50× slower than the cache, the whole average is dominated by the misses. This is the counterintuitive heart of caching: the last few percent of hit rate matter most, because they're cutting the expensive path, not the cheap one. A staff-level corollary: cache what's slow and hot, not merely hot — a 95% cache that keeps missing your one expensive query can lose to a 90% cache that catches it.

And there's a second win hiding in the same number. A 90% hit rate means the database only ever sees the 10% that miss — a 10× cut in read load. Caching doesn't just make reads fast; it shields the tier you can least afford to overload. The database is the hardest thing in your system to scale — it holds the truth, it's stateful, it doesn't shard by wishing. A cache in front of it soaks up the repetitive reads so it can spend its scarce capacity on writes and on the queries that genuinely need it. That's why, when a read-heavy system starts to strain, a cache is almost always the first thing you reach for, long before you shard.

A cache is a tool with a shape, and forcing it onto the wrong workload makes things worse, not better. It loses when:

• The workload is write-heavy. If a value changes almost as often as it's read, you spend more effort keeping the cached copy correct than you ever save on reads — the cache becomes pure overhead, sometimes a net loss.
• There's no locality. If reads are spread uniformly across a huge keyspace — the 1% case from the sim — the cache thrashes, evicting entries before they're ever reused, and helps nobody. No hot keys, no point.
• The data must be exactly current. A cache serves copies, and copies can be stale. For a value that has to be right to the instant — an account balance at the teller, the remaining seats as you click buy — a cached read can hand back a lie. Sometimes you cache anyway and accept a tiny staleness window; sometimes you can't.
• The data is already small or fast. If the working set already fits in the database's own memory, or reads are already sub-millisecond, a separate cache just adds a network hop and a second copy to keep honest.

And even when a cache clearly wins, it isn't free: it's another system to run, more memory to buy, and it drags in the genuinely hard problem — keeping the copy from lying to you. Deciding when a cached value is stale, and what to do about it, is famously "the second-hardest problem in computer science," and it gets its own lesson soon. Reach for a cache when reads dominate, a few keys are hot, and a little staleness is survivable — and think twice everywhere else.

A few caching beliefs feel obviously true and quietly cause trouble:

• "Add a cache and everything gets fast." → Only if the traffic has locality and your hit rate is high. On a uniform, write-heavy, or already-fast workload, a cache adds a hop and a consistency headache for little or no gain. The cache is a bet on skew — if the bet's wrong, you lose.
• "Caching makes the database faster." → It doesn't touch the database's speed at all. It makes the database do less work by answering the repeat reads elsewhere. Same database, fewer trips to it.
• "A higher hit rate is always the goal." → Usually, but not blindly — a slightly lower hit rate that catches your expensive queries beats a higher one that only catches cheap ones. What you're really minimizing is time spent on misses, not the miss count.
• "If a little caching is good, cache everything." → Cache the hot, slow, rarely-changing reads. Caching cold or fast-changing data just fills memory with entries that either never get reused or go stale immediately — cost with no payoff.

• Caching keeps a copy of the answer somewhere fast, so repeat reads skip the slow trip down the latency ladder (RAM is ~1,500× faster than an SSD read, ~100,000× faster than a disk seek). It avoids the work rather than speeding it up.
• It works because of locality. Real traffic is skewed (Zipf / 80-20) — a few hot keys get most of the requests — so a tiny cache captures a huge share of reads. No skew, no win: the same cache goes from ~78% to ~1% hit rate on uniform traffic.
• Hit rate is the whole game, non-linearly. effective latency = hit×cache + miss×db, dominated by the miss term — so 90%→99% can be a ~4× speedup. And a high hit rate cuts the database's read load by the same factor, shielding the tier you can least afford to overload.
• It's not universal. Write-heavy, no-locality, must-be-current, or already-fast workloads don't benefit — and every cache adds memory, a moving part, and the staleness problem.

Next — Caching Patterns: Cache-Aside, Read-/Write-Through, Write-Behind: now that you know a cache is worth it, the immediate question is who's responsible for filling it and keeping it honest — the application, or the cache itself — and each answer trades performance for a different flavor of staleness or data-loss risk.