Checkpoint: Storage Systems

You opened this section with a simple-sounding question — where do I put the bytes? — and the comfortable assumption that "the cloud" is one big disk in the sky.

Since then you've learned it is nothing of the kind. You met three abstractions — a raw block device, a shared file system, an object store — each with its own semantics and its own price. You built the inside of an object store and found it splits into two planes, a tiny metadata map and a dumb fleet of bytes. You watched one master run a cluster of thousands by staying out of the data path — 45 bytes of metadata through it, 64 MB of data around it. You learned to buy durability two ways — ×3 copies or 1.4× of parity — and to move a four-gigabyte object over hotel Wi-Fi one resumable chunk at a time. You fanned one video master into a ladder of renditions a player adapts to, segment by segment. And you learned that data has a temperature, and that the cheapest tier is the slowest and costliest to read.

Seven lessons. A whole warehouse of tools — block, file, object; replication and erasure coding; chunks, segments, and tiers.

And every one of them is an answer to the same decision.

Here is the one idea the whole section was built on.

There is no "storage." There is only the right store for this byte.

Every tool you learned answers one of four questions about a specific piece of data — and you cannot maximize all four at once:

• How is it accessed? A raw device (block), shared paths with locks (file), or HTTP-addressable blobs (object). Three abstractions, three prices.
• How durable must it be, and at what price? Full copies (replication, ×3, cheap reads) or data + parity shards (erasure coding, ~1.4×, costly reads). Same survival, a different bill.
• How does it move? A byte too big for one request is cut into chunks — which makes the transfer resumable, parallel, and content-addressable for dedup.
• How hot is it? Match the tier to its temperature — hot on fast storage, cold in the archive as it ages — because cheap-to-store is costly-and-slow-to-read.

That is the decision. And underneath every lesson there is a single recurring move — the trick that makes each of these systems actually scale:

Separate the map from the territory. Keep the metadata small, indexed, and in the fast path; keep the data huge, streamed, and out of it.

It is the same idea wearing different clothes in every lesson. The GFS master is the map (45 bytes) and the chunkservers are the territory (64 MB). An object store's metadata plane is the map and its data plane is the territory. A content hash is a map that names the bytes it points at, which is how chunking dedups. A lifecycle index stays hot and small while the data it tracks ages into cold archive. Small hot map, huge cold territory — find that split, and a system that shouldn't scale suddenly does.

Read this table slowly. It is the section, restated in one line each — and the point is not the seven rows, it's that they are all the same decision.

the lesson	…is really deciding
Block vs File vs Object Storage	What shape does this data want to be accessed as? Raw device, shared file paths, or HTTP blobs — three abstractions, three prices. Match the shape, not the sticker.
Design an S3: Blob Storage as a System	How do you build the object store? Split it — a tiny metadata plane (key→location) and a dumb data plane (bytes). The map, and the territory.
Distributed File Systems	How does ONE master run a petabyte cluster? By staying out of the data path — control (45 B) through it, data (64 MB) around it. The map is small enough to hold in RAM.
Erasure Coding vs Replication	How do you buy durability, and how much do reads cost? Copies (×3, cheap reads) or parity + math (1.4×, costly reads). Same survival, different bill.
Media Chunking	How do you move a byte too big for one request? Cut it into parts — resumable (re-send only what's missing), parallel, and content-addressed so identical bytes dedup.
Video Transcoding Pipelines	One master, every screen? Fan it out into a bitrate ladder; the player adapts per segment; the build pipeline is embarrassingly parallel.
Tiered & Cold Storage	Where does a byte live as it ages? Match the tier to the temperature — and remember the cheapest tier is the slowest, costliest one to read.

Every row is the same shape: name the data's properties — access, durability, size, temperature — then send it to the store that fits, and keep the small hot map separate from the huge cold territory. The whole art of the section is refusing the one-size answer ("just put it on S3") and matching each byte to where it actually belongs.

Enough theory. Here are four pieces of real data, each with a hard constraint, and the whole warehouse you've spent seven lessons filling.

A warning before you start: not one of the four answers is the one you'll reach for first. Each shape opens with the safe-sounding instinct — a big disk, "just use S3," replicate to be safe, keep it cheap — and each of those instincts is a trap the section named out loud. Pick a store, watch it fit or break, and find the call that actually works.

The 4 GB video master. Your instinct is a big block volume kept hot. But block is a single-machine raw disk — it can't be a globally served object, and one file fits no screen. The answer walks the whole section: chunked resumable upload in, transcode to a bitrate ladder, serve renditions from a CDN, and age the master to Glacier.

The 7-year compliance logs. Your instinct is Standard ×3, "to be safe." But that's 23× overpriced for data no one reads, tripled. The answer is Glacier Deep Archive with erasure coding — cheapest storage, cheap durability, and a 12-hour restore that doesn't matter for write-once data.

The shared home directory. Your instinct is S3, because it's cloud-scale. But object storage has no rename, no locks, no partial writes — it can't be POSIX. The answer is a network file system (EFS/NFS): shared paths, locks, in-place edits, mounted by many machines.

The petabyte scanned nightly. Your instinct is one giant file on one big machine. But no disk holds a petabyte and one box can't feed hundreds of readers. The answer is a distributed file system — 64 MB chunks spread ×3 across chunkservers, the master out of the data path, so the scan runs massively parallel.

⭐ Look at the shape of the four answers: an object store + pipeline, the cheapest cold tier, a file system, and a distributed file system. Not one was "put it all on S3." Each byte went to the store that matched how it's accessed, how durable it must be, how it moves, and how hot it is — and every one kept the small hot map separate from the huge cold territory.

Eight calls. Every wrong option is a trap this section warned you about by name — lifted straight from the seven lessons' Mental-Model Corrections. If one of them looks obviously right to you, that's worth knowing before it's a production bill.

Pass mark: 6 of 8.

Concretely — things you couldn't do seven lessons ago:

• Pick block vs file vs object from the access pattern, not the price tag — raw disk, shared POSIX paths with locks, or HTTP blobs — and never again answer "just use S3" for a shared home directory.
• Design an object store as two planes — a small hot metadata map and a dumb cheap data fleet — and recognize that split everywhere it recurs.
• Explain why one master scales a petabyte cluster — it's out of the data path; 45 bytes of metadata through it, 64 MB of data around it.
• Choose replication vs erasure coding by temperature and size — ×3 copies for hot/small, ~1.4× erasure coding for cold/large — knowing EC trades cheap storage for costly reads and rebuilds.
• Move a giant object politely — multipart + resumable (re-send only the missing parts), and content-defined chunking so an inserted byte doesn't nuke your dedup.
• Turn one master into every screen — a bitrate ladder of keyframe-aligned segments that the player adapts to, built by an embarrassingly-parallel pipeline.
• ⭐ Tier by access pattern, not wishful thinking — cheapest storage for data that stays cold; and price in the retrieval latency and the one-time storm before you send hot data to Glacier.
• ⭐ Find the map/territory split in any storage system — and know that keeping metadata small, indexed, and hot is what lets the data underneath it grow without bound.

• ⭐⭐⭐ The law: there is no "storage," only the right store for this byte. Four questions decide it — how it's accessed (block/file/object), how durable at what price (replication vs erasure coding), how it moves (chunking), how hot it is (tiering) — and you can't max all four.
• ⭐⭐⭐ The recurring move: separate the map from the territory. Small hot metadata in the fast path; huge cold data out of it. The GFS master, S3's two planes, a content hash, the lifecycle index — all the same trick, and it's what makes each system scale.
• ⭐ Match the abstraction to the access. Object storage can't be POSIX; block is single-attach; a distributed file system is for big sequential scans, not small-file random access.
• ⭐ Durability has two prices. ×3 replication (cheap reads) vs ~1.4× erasure coding (cheap storage, costly reads) — hot/small replicates, cold/large erasure-codes.
• Move big things in chunks, and cut on content. Resume = re-send the missing parts; fixed-size chunking dies on an inserted byte (0% dedup), content-defined survives (~98%).
• Data has a temperature; the cheapest tier is the slowest to read. Tier by access pattern; hot data belongs hot, and a Glacier restore is a 12-hour, per-GB-billed event, not a GET.

That closes Storage Systems — how the bytes are shaped, spread, moved, and aged. You've now built the whole data layer: how it's modeled, how it's stored on an engine, how it's replicated and scaled, how it stays correct under concurrency, and where it physically lives. Next is §8 — Specialized Indexes & Sketches: what to do when an ordinary B-tree index simply can't answer the question — is this in the set, what's near this point, does this text match — and the beautiful, slightly magical structures that can.