Distributed File Systems

One File, a Thousand Machines

In Design an S3 you built the object store's data plane — a fleet of nodes packing bytes into big chunks — and I promised it had a famous lineage. This is it. Long before S3, Google had a problem that no filesystem in existence could handle: store the entire web crawl — single files of many terabytes — across thousands of cheap machines that were failing all the time (with that many disks, something is always broken), for a workload that was almost entirely huge sequential reads and appends. Their answer, the 2003 Google File System (GFS) paper, is the blueprint that HDFS, Colossus, and every big-data filesystem since has followed.

Your laptop's filesystem — ext4, NTFS, APFS — is the wrong tool for this, and not by a little. It assumes one disk on one machine, small files, random edits, and that hardware basically works. GFS assumes the opposite of every one of those: files bigger than any disk, machines that die as a matter of routine, and a workload of append-and-scan. So it makes three moves your laptop never would — split every file into big chunks, replicate each chunk across machines, and put one master in charge of the map. Here's the whole anatomy on one page; the rest of the lesson walks it.

The anatomy of a distributed file system in the GFS style. At the top, a file, /crawl/2024.warc, is split into big chunks c0, c1, and c2, colour-coded — deliberately large, 64 megabytes each, and each replicated three times. On the left, in violet, the single MASTER holds the metadata map in RAM: c0 lives on cs1, cs2, cs3; c1 on cs2, cs4, cs5; c2 on cs1, cs3, cs5 — and it never holds a byte of your data; it knows only where each chunk lives, the map and not the territory. Along the bottom, five chunkservers hold the actual bytes, each showing its coloured chunk chips, so you can see every chunk colour repeating exactly three times across the machines — that repetition is the replication. The bottom line: each chunk lives on three machines, and if one is lost the master re-replicates it from a surviving copy.

Chunks and Chunkservers

A file is too big for one disk, so GFS chops it into fixed-size chunks — 64 MB each (HDFS uses 128 MB blocks; same idea). Each chunk is just a plain file on a chunkserver (an ordinary Linux box with disks), and — because machines die — each chunk is stored on three chunkservers, spread across different racks. Here it is on a real (miniature) cluster:

$ put /crawl/2024.warc   → 17 chunks × 3 replicas

$ ls the chunkservers — the DATA lives here, spread + replicated:
    cs1/ : 10 chunk files   cs2/ : 8   cs3/ : 11   cs4/ : 9   cs5/ : 13

One file became 17 chunks, each written to 3 of the 5 chunkservers — so every machine holds a slice of the file, and every chunk survives two machine deaths.

Why are chunks so big? It's a deliberate trade, and it's about metadata, not disk. A bigger chunk means fewer chunks for the master to track, fewer times a client has to ask the master "where's the next piece?", and long sequential reads/writes that keep a disk streaming. (And no, a 64 MB chunk does not waste 64 MB on a 1 KB file — a chunk is a normal file that's allocated lazily, so a 1 KB object uses 1 KB. The 64 MB is a ceiling and the unit of bookkeeping, not a minimum size on disk.) This is the same instinct as Design an S3's packed extents — keep the number of things you have to track small, even as the bytes grow without limit.

The Master Holds the Map, Not the Territory

Someone has to know which chunks make up /crawl/2024.warc, and which chunkservers hold each chunk. That someone is a single master — one machine that holds all the metadata, in RAM:

$ ask the master (this is ALL it holds):
    /crawl/2024.warc → 17 chunks
      …_c0 → replicas on ['cs4', 'cs5', 'cs2']
      …_c1 → replicas on ['cs5', 'cs2', 'cs1']
      …_c2 → replicas on ['cs5', 'cs4', 'cs3']
      …

That's it. The master is the namespace (which chunks belong to which file) and the location map (which chunkservers hold each chunk). It is the map, not the territory — the master never stores a single byte of your file data. That data lives only on the chunkservers.

Keeping all metadata in one machine's RAM is what makes the master fast — a lookup is a memory access, not a disk seek. But RAM is volatile, so the master's metadata is made durable the classic way: every change is written to a replicated operation log, with periodic checkpoints, so a master that crashes and restarts rebuilds its entire state by replaying the log. (In HDFS the master is the NameNode, the chunkservers are DataNodes, and a standby NameNode takes over on failure.) A master crash costs you availability for a moment — it never costs you a byte of data, because the data was never on it.

The Trick: the Master Steps Out of the Data Path

Now the obvious objection: one master for a cluster of thousands? Isn't that a bottleneck — every read and write funnelling through a single machine? It would be, if the data went through the master. It doesn't. This is the single most important idea in the whole design.

Watch a read. The client wants chunk c2 of a file:

① client → master:  "where is chunk c2?"
② master → client:  ['cs5','cs4','cs3']   (45 bytes of metadata — the master's whole job)
③ client → cs5 DIRECT:  read the 64 MB chunk   (the master never sees the bytes)

The master handled 45 bytes of metadata; the chunkserver moved the whole 64 MB of data — directly to the client, bypassing the master entirely. The client even caches that location, so its next reads of the same chunk don't touch the master at all. So the master does a tiny amount of thinking (where is it?) while the chunkservers do all the heavy lifting (move the bytes).

That ratio is the whole answer. At real scale it's kilobytes of metadata through one master vs petabytes of data flowing client↔chunkserver in parallel. A single master isn't a bottleneck because it was never in the fast lane — it stands at the side of the road handing out directions. Separate the control plane (the master) from the data plane (the chunkservers), and one brain can run a cluster of thousands. (Writes work the same way: the master hands out a lease to one replica — the primary — that orders the mutation, and the data itself is pushed chunkserver to chunkserver in a pipelined chain, never through the master. GFS's signature operation is atomic record append, because its whole workload is many producers appending to shared files.)

Why a single master scales: it is not in the data path. Three nodes — a master at the top, a client at bottom-left, a chunkserver at bottom-right. The control flow, drawn as thin amber arrows between the client and the master: the client asks where is chunk c2, and the master answers cs5 — about 45 bytes of metadata, the master's entire job — with a red note that the bytes never come through here. The data flow, drawn as one fat blue arrow straight between the client and the chunkserver: the client reads the 64-megabyte chunk directly from the chunkserver. The bottom line: 45 bytes of metadata pass through the single master while 64 megabytes — and at scale, petabytes — flow directly between client and chunkserver, which is exactly why one master can run a cluster of thousands of machines.

Drive it yourself. Read a chunk and watch the master grey out of the data path; then kill a chunkserver and watch what happens next.

Drive a GFS cluster: one master, five chunkservers, a file in three replicated chunks. Read a chunk and watch the master hand out a location — a few dozen bytes of metadata — then grey out as the data flows client↔chunkserver directly, never through it. Watch the counter: bytes through the master vs bytes moved by the chunkservers. Then kill a chunkserver and watch the cluster heal itself — the master notices the missing heartbeat and re-replicates every now-under-replicated chunk from a surviving copy, back to ×3. No data lost, no human paged.

When a Machine Dies

In a cluster of thousands of commodity disks, a machine dying isn't an emergency — it's Tuesday. So GFS is built to treat failure as the normal case, and to heal without a human. Every chunkserver sends the master a periodic heartbeat. When one goes silent, the master knows its chunks just lost a replica — they're now under-replicated — and it acts:

cs2's heartbeat stops → master finds 8 chunks now under-replicated (only 2 copies)
master re-replicated 8 chunks from survivors → all back to ×3
every chunk at 3 replicas again?  True

The master picks a surviving replica of each affected chunk, tells a healthy chunkserver to copy it, and the cluster is back to ×3 — no data lost, no page fired. This is why the replicas are spread across racks in the first place: a whole rack or switch can fail and no chunk loses all three copies. Machine death is designed in, not patched around — which is exactly the mindset from Replication Topologies, now running continuously and automatically across a storage cluster.

Where the Single Master Breaks

So is the single master free of all limits? No — and knowing its real ceiling is the mark of actually understanding the design. The master isn't a bottleneck for data (it's out of that path), but it is the bottleneck for metadata: every file and chunk is an entry in its RAM, and every open/create/lookup is an operation it must serve. At Google's scale, the number of files GFS could hold and the rate of metadata operations eventually ran into one machine's memory and CPU.

The fix is to do to the metadata what GFS already did to the data: distribute it. Google's GFS successor, Colossus, replaces the single master with a distributed metadata layer (sharded across many servers, itself stored in a Bigtable-like store) — so the metadata scales horizontally, the same way the chunkservers always did. HDFS reached for the same idea with NameNode federation (multiple NameNodes, each owning part of the namespace). The lineage is clean: GFS (2003) → HDFS → Colossus — each step keeping the chunks-and-replicas data plane and pushing the metadata plane further out, from one master's RAM to a distributed store.

What to Remember

⭐ A distributed file system spreads a file as replicated chunks across failure-prone machines. Files → big chunks (64 MB), each on ×3 chunkservers across racks; a single master holds the metadata map (file → chunks → locations) in RAM and never holds a byte of data.
⭐⭐ The master is not in the data path — that's why one master runs a cluster of thousands. A read is a tiny control message to the master for a location (measured: ~45 bytes), then the bytes flow client↔chunkserver directly (64 MB → petabytes at scale). Control plane vs data plane, separated.
The cluster heals itself. Heartbeats detect a dead chunkserver; the master re-replicates its under-replicated chunks from survivors, back to ×3 (measured: cs2 dies → 8 chunks → ×3). Failure is the normal case, not the emergency.
The master's real limit is metadata, not data — its RAM and op rate. That ceiling is why GFS → HDFS → Colossus pushed the metadata itself into a distributed store.

One thing we waved past: keeping three full copies of every chunk is expensive — it triples your storage bill for durability. Is there a cheaper way to survive machine death than paying 3× for the bytes? There is, and it's a beautiful piece of math — Erasure Coding vs Replication, next, on durability per dollar.