Design an S3: Blob Storage as a System

In Block vs File vs Object Storage you learned what object storage is: a flat key → an immutable blob, reached over HTTP, that scales without limit and costs the least per gigabyte. It's where the internet keeps its photos, videos, backups, and data lakes. Now the obvious question — the one that turns a user into an engineer: how do you actually build that?

Write the spec down and it looks almost impossible. You need to store trillions of objects, from a few bytes to a few terabytes each. You need eleven nines of durability — lose one object in a hundred billion per year and you've failed. You need to find any object instantly by a flat string key, and list millions of them by prefix. It has to be the cheapest storage you offer. And it has to keep doing all of that while machines and disks die constantly underneath it.

It turns out the whole thing rests on one decision — a decision so clean that once you see it, every other piece of the design falls into place. Here's the shape of the answer; the rest of the lesson earns it:

Start with the obvious design, because watching it fail is what teaches you the real one. You have a File abstraction already (that's the middle rung from last lesson) — so why not just save each object as a file? cat.jpg becomes a file named cat.jpg; done. It works beautifully for a thousand objects, and it is unbuildable for a trillion. Two things break, and both are measurable.

Wasted space. A filesystem allocates space in blocks — 4 KB at a time. A 100-byte object still consumes a whole 4 KB block. Store fifty thousand tiny objects and measure the disk:

ONE FILE PER OBJECT   (50,000 blobs × 100 B):
  logical data ....  4.8M     (what you actually stored)
  ACTUAL disk .....  197M     ← each 100 B blob rounds up to a 4 KB block  (~41×)
  inodes ..........  50000    ← 50,000 filesystem metadata entries, for 5 MB of data

197 MB of disk to hold 4.8 MB of data — a 41× blowup — plus fifty thousand inodes, the filesystem's own per-file records. Now multiply by a trillion. The disk waste alone would bankrupt you, and no single filesystem on Earth can track a trillion inodes or list a directory with a billion entries in it.

The fix for the space is already visible if you just pack the objects — append them all into one big file, back to back:

PACKED INTO ONE EXTENT (append every blob to one big file):
  ACTUAL disk .....  4.8M     ← no per-object block rounding; 1 inode, not 50,000

Same data, 4.8 MB and one inode. So the bytes want to live packed into big files. But now you've created a new problem: if cat.jpg's bytes are somewhere in the middle of a 100 GB packed file, how do you ever find them again? That question is the whole design.

Here is the one decision. Split the system into two independent planes — because the two things you're storing have completely different shapes and want completely different machines.

The data plane — the bytes. These are big and simple. You pack incoming objects into large append-only files — call them extents (Facebook's Haystack calls its ~100 GB versions needle files) — spread across thousands of cheap storage nodes. Appending is the only write; immutability (from last lesson) makes that natural — you never edit an object, so you never have to seek back and overwrite. An object's bytes live at a location: (volume, offset, length). This plane is the easy part — it's just fast sequential writes to big files, and it scales by adding more nodes.

The metadata plane — the index. This is the map from key → location: cat.jpg → (volume 3, offset 40k, length …), plus the size, content-type, ETag, version, and any user metadata. These records are small but there are trillions of them — one per object — and they must answer two questions fast: where is object X? and list the objects with prefix P? So the metadata plane is a sharded, sorted key-value index (partitioned by bucket and key), living on its own fleet. This plane is the hard, interesting part — trillions of tiny records and range queries — and it scales by sharding the index.

⭐ The whole design is that split. The bytes are easy; the index is the work — and because the two have opposite shapes (huge-and-sequential vs tiny-and-numerous), you let them scale on separate fleets, independently. You saw this in miniature: on a single MinIO node, one object is stored as two files — xl.meta (401 bytes: the metadata record — version, ETag, and where the data is) and part.1 (1,200,064 bytes: the actual object bytes). Even one node splits the map from the territory. A real object store just spreads each plane across thousands of machines.

With the two planes in place, every request is a short trip through them. A client never touches either plane directly — it hits a front door (a load balancer and an API tier) that orchestrates the rest.

Writing an object (PUT) goes data-first, then metadata:

1. The front door receives the key and the bytes.
2. Data plane: pick a volume with room and append the bytes to it, durably (replicated or erasure-coded across nodes — next section). Get back the location (volume, offset, length).
3. Metadata plane: record key → location (plus size, ETag, metadata) — atomically.
4. Return 200 OK with an ETag.

The order matters, and it's what gives you strong read-after-write consistency: the object is invisible until step 3 commits. If the write dies after step 2, you have some orphaned bytes in an extent (garbage-collected later) but never a key that points at nothing. Notice you did not need a distributed transaction across the two planes — write the immutable bytes first, then commit one small metadata record. (Objects too big for one request use multipart upload — split into parts, upload them in parallel, then commit one metadata record that stitches them together; it also gives you resumable uploads, which is the whole of Media Chunking two lessons from now.)

Reading an object (GET) runs the planes the other way:

1. The front door receives the key.
2. Metadata plane: look up key → location (or return 404 if there's no record).
3. Data plane: read length bytes at offset in that volume, and stream them back.

One index lookup, one sequential read. That's the entire hot path of S3.

Three details hang off the split, and each one lands squarely in one plane.

Durability lives in the data plane. Eleven nines doesn't come free — every object's bytes are stored redundantly across independent failure domains (different nodes, racks, availability zones), so no single disk or machine death loses anything. There are two ways to buy that redundancy — replication (keep N whole copies) and erasure coding (split into data + parity shards) — and the trade between durability and dollars is the entire next-but-one lesson, Erasure Coding vs Replication. For now: the data plane is redundant, by one of those two schemes.

Listing lives in the metadata plane. LIST bucket/2024/ — "give me every key under this prefix" — is a range scan, which only works if the index is sorted by key. That's why the metadata store is a sorted structure (a B-tree or LSM-tree), and why LIST is a metadata operation that never touches the bytes at all. The data plane has no idea what a "prefix" is; it only knows offsets.

Deletes are lazy. You can't cheaply punch a hole in the middle of a 100 GB append-only extent. So a DELETE just writes a tombstone in the metadata — the object vanishes from every read and list immediately — while the bytes sit dead in the extent until a background compaction job eventually rewrites the live objects into a fresh extent and reclaims the space. Deleting is a metadata edit now, a data cleanup later.

You now have every piece: pack the bytes into append-only extents, index them with a sharded key→location map, make the bytes redundant, and orchestrate PUT and GET through a front door. So build it — and, more usefully, try to build it wrong.

Below, make the three decisions that define an object store — how to store the bytes, how to index them, and how durable to be — and watch a live scoreboard price each choice. Then fire a PUT and a GET and watch the request flow through your design: the bytes landing in the data plane, the key → location record landing in the metadata plane, the object appearing only when that commits. Pick "a file per object" and watch the disk explode; pick "scan a directory" and watch listing die; pick "one copy" and watch a disk failure eat your data. Only one combination is buildable — and finding it is the design.

• ⭐⭐ An object store is two planes, split because the two things it stores have opposite shapes. The data plane holds the bytes — packed into big append-only extents on many nodes; huge and simple, scales by adding nodes. The metadata plane holds the key → location index — trillions of tiny records, sharded and sorted; small and numerous, scales by sharding.
• ⭐ The bytes are easy; the index is the work. One file per object is unbuildable — 41× disk waste and a filesystem entry per object (measured: 197 MB and 50,000 inodes for 4.8 MB). Pack the bytes; keep a separate index.
• PUT is data-then-metadata; GET is metadata-then-data. The object is invisible until the metadata commits — that's strong read-after-write, with no cross-plane transaction, thanks to immutability.
• Each concern lands in one plane: durability in the data plane (replicate or erasure-code, → Erasure Coding vs Replication); LIST in the metadata plane (a sorted range scan); DELETE is a metadata tombstone now, a data compaction later.

You built the object store as a metadata plane over a data plane — but that data plane, the fleet of nodes packing bytes into big chunks and keeping them alive as machines die, is a whole system of its own with a famous lineage. That's next — Distributed File Systems — where you meet the chunks-and-masters design (GFS and its descendants) that sits underneath the bytes.