The Memory Hierarchy: Disk, RAM, CPU Cache

Here's a question that sounds trivial and isn't: why is a cache fast?

"Because it's in memory" is the usual answer, and it's not wrong — but it doesn't explain why your database, also largely in memory, can still be slow, or why adding RAM sometimes fixes nothing. The real answer is a two-part story that, once you have it, makes an enormous amount of system design suddenly obvious: caching, indexes, why databases are shaped the way they are, why "just add a bigger machine" hits a wall.

In the last lesson you met the latency ladder — the fact that different operations span an absurd range of speeds. This lesson explains why that ladder exists (it's not an accident, it's a law) and why it's the best deal available (a gift called locality). Get this, and "why is a cache fast?" stops being something you memorize and becomes something you can derive.

In this lesson: the tiers and their real numbers · the iron trade-off that forces the hierarchy · the principle of locality that makes it work · and the effective-latency math behind every cache you'll ever design.

Scope: we cover why the hierarchy works here; how access patterns change everything is the next lesson (Sequential vs Random Access), and caching as a system-design pattern (cache-aside, eviction, CDNs) gets its own section later (Caching). Today: the ladder itself.

Every computer would love one kind of memory: fast, huge, cheap, and permanent. No such thing exists, and it isn't an engineering failure to be fixed next year — it's a fundamental tension of physics and economics:

• The memory that's fastest (SRAM, used for CPU caches) is enormous per bit, power-hungry, and expensive — so you can only afford a tiny amount, physically close to the CPU.
• The memory that's cheapest per bit (spinning disk, flash) is slow — so you can afford oceans of it, but every access costs dearly in time.

Fast is small and expensive. Big is slow and cheap. You cannot have all three.

Since no single memory wins, computers do the only sensible thing: they stack tiers. A sliver of blazing-fast memory right next to the CPU, backed by a larger, slower tier, backed by a much larger, much slower one, all the way down to disk. Each layer is a compromise, and the stack is the escape from having to pick just one point on the trade-off. That stack is the memory hierarchy.

Here is the actual hierarchy on a modern machine. Don't memorize the digits — burn in the orders of magnitude and the shape:

Read the extremes: an L1 hit is ~0.3 nanoseconds; a disk seek is ~10 milliseconds — that's a thirty-million-fold difference in the same machine. Two facts jump out and both matter for the rest of the course:

1. The RAM→disk cliff. RAM is ~100 ns; a spinning disk is ~10 ms — a 100,000× gap. This is the reason "is it in memory or on disk?" is one of the most consequential questions in system design.
2. The volatility line. Everything at RAM and above is volatile — pull the power and it's gone. Only SSD/HDD persist. That single line is why databases exist, why fsync matters, and why "it's in the cache" is never the same as "it's saved."

Nanoseconds and milliseconds are both just "small," so the brain flattens them. Fix that by multiplying every latency by one billion — turning nanoseconds into seconds:

Sit with that. If grabbing a value from L1 cache feels like one second, then going to RAM is a coffee break, hitting an SSD is a long weekend, and reaching for a spinning disk is a season of the year. When an engineer says "just read it from disk," they're proposing, in human terms, to wait four months instead of one second. That is why the whole edifice of caching, indexing, and in-memory data structures exists — to avoid the long trips. You'll feel this scale under every performance decision in this course.

Stacking tiers only helps if the tiny fast tier manages to satisfy most requests — otherwise you're always falling through to the slow bottom and the hierarchy buys nothing. So why does a cache holding maybe 1% of your data catch 95% of your accesses? Because real programs are not random. They obey the principle of locality:

• Temporal locality — if you used it recently, you'll probably use it again soon. The user whose profile you just loaded is about to load their settings; the trending video is watched a million times an hour. Keep recent things hot.
• Spatial locality — if you used something, you'll probably use its neighbors next. Iterating an array, reading adjacent database rows, scanning a file. This is so reliable that hardware bets on it: memory moves in 64-byte cache lines, not single bytes — when you touch one byte, the CPU drags its 63 neighbors along, wagering you'll want them.

Locality is the quiet miracle that makes the hierarchy pay off. A small fast tier + locality means the common case is fast and only the rare case is slow. That is the real answer to "why is a cache fast?" — not because the cache is magic, but because your access patterns let a small amount of fast memory do most of the work.

Locality + a fast tier gives you a number you can actually compute — and it's the single most useful formula in performance work:

Effective latency = (hit rate × fast tier) + (miss rate × slow tier)

Put real numbers in. Say a RAM cache (100 ns) sits in front of a disk (10 ms), and thanks to locality you hit the cache 95% of the time:

0.95 × 100 ns + 0.05 × 10,000,000 ns ≈ 500,000 ns = 0.5 ms

Always-disk would be 10 ms. The cache made it ~20× faster — while holding only a fraction of the data. Push the hit rate to 99% and it's ~100 µs, a 100× win. This is the entire economic argument for caching, and it's why you'll meet "hit rate" in nearly every performance conversation ahead.

Notice the cruelty in the formula too: the misses dominate. Because the slow tier is so much slower, even a small miss rate drags the average up hard. A cache that drops from 99% to 90% hit rate doesn't get 10% slower — it gets ~10× slower. That asymmetry is why cache tuning is worth real effort, and why a cold cache (0% hits) can take a system down. (The design patterns for keeping hit rates high — eviction, warming, invalidation — are their own section, Caching.)

Now drive the formula yourself. Pick a fast tier and a slow tier from the hierarchy, then drag the hit rate and watch two things move in real time: a live stream of requests (green = hit the fast tier, red = fell through to the slow one) and the effective latency readout.

Try this first: set RAM as fast and HDD as slow, then drag the hit rate from 50% up to 99%. Watch the effective latency collapse — and notice how much of the drop happens in the last few percent. That curve is the reason engineers obsess over the last points of hit rate.

The hierarchy is itself the trade-off, made physical — and using it well means respecting what each tier costs you, not just what it buys.

What the fast tiers buy: speed, obviously. What they bill: they're small (so you must choose what to keep hot — that's eviction), they're expensive (RAM at ~$3–5/GB vs disk at ~$0.02/GB is a hundred-fold cost difference, a real line item on a cloud bill), and they're volatile (so they can never be your source of truth — a cache is a fast copy, never the original).

The classic mistake this creates: treating the cache as the truth. Data written only to RAM and not persisted is one power-blip from gone — which is exactly why every serious system keeps the durable copy on disk and treats the fast tiers as accelerators, not vaults. "It's in the cache" and "it's saved" are different sentences, and confusing them has ended companies. The skill is using each tier for what it's good at: disk for truth, RAM and caches for speed, and a hit rate high enough that the truth is rarely consulted directly.

A prediction, then a check — do the prediction before reading the answer:

You have a service doing 10,000 reads/second, all currently from a database on SSD (~150 µs each). You add a RAM cache (~100 ns) and measure an 80% hit rate.

1. Predict: roughly what's the new effective latency per read? Did it help much?
2. Now compute it: 0.80 × 100 ns + 0.20 × 150,000 ns = ?
3. Then: you tune the cache and reach 99% hit rate. Recompute. How much better than 80%?

Check: 80% gives ≈ 30 µs (150,000 × 0.2 ≈ 30,000 ns) — already ~5× better than always-SSD, but the misses still dominate. 99% gives ≈ 1.6 µs — another ~20× on top. The lesson in the numbers: at high hit rates, the misses are almost the entire cost, so the last few percent of hit rate matter more than the first fifty. If your intuition said "80% is basically done," the math just corrected it — that correction is worth real money in production.

• "RAM and disk are both just 'storage.'" → ~100 ns vs ~10 ms is a 100,000× gap, and RAM is volatile while disk persists. They are not interchangeable; treating them as such is behind a huge share of slow, or lossy, systems.
• "Just add more RAM and it'll be fast." → Only if your working set fits and has locality. Random access across a dataset far larger than RAM still misses constantly — more RAM with no locality buys little.
• "The cache is something I turn on and configure." → The CPU caches (L1/L2/L3) are automatic hardware — you influence them only through how you access memory (next lesson). The caches you design (Redis, CDN) are the application-level version, and they're a later section.
• "A higher hit rate is nice-to-have." → Because misses dominate the effective-latency formula, the jump from 90% to 99% is often a 10× speedup. Hit rate isn't a vanity metric; it's frequently the performance lever.
• "If it's in the cache, it's saved." → Caches are volatile copies. The durable truth lives on disk (in a database). A cache accelerates reads; it does not remember.

• A hierarchy must exist because fast memory is small and expensive while big memory is slow and cheap — no single tier wins, so computers stack them.
• Learn the shape and orders of magnitude: registers ~0.3 ns → L1 ~1 ns → RAM ~100 ns → SSD ~150 µs → disk ~10 ms. The RAM→disk cliff is ~100,000×, and the volatility line (RAM and up = lost on power-off) is why disk holds the truth.
• Locality (temporal + spatial) is why a tiny fast tier works — real programs reuse recent and nearby data, so a small cache catches most accesses. That's the true answer to "why is a cache fast?"
• Effective latency = hit×fast + miss×slow. A 95% RAM-over-disk hit rate is ~20× faster while caching a fraction of the data — and because misses dominate, the last few points of hit rate matter most.
• A cache is a fast, volatile copy — never the source of truth. Disk for truth, fast tiers for speed.
• Next — Sequential vs Random Access: Why Order Matters: the same tiers behave wildly differently depending on how you touch them — the physics behind WAL, LSM trees, and Kafka.