Partitioning vs Sharding

The whole of Scaling Reads worked because of a single property: a read can be answered by any copy of the data. An index is a copy shaped for lookups; a replica is a copy on another box; a cache is a copy in front; a materialized view is a copy precomputed. Every §4 lever was a way to make or place a copy, and the checkpoint ended on the sentence that opens this section: no copy can absorb a write.

Here's why that sentence is load-bearing. A write has exactly one authority — the owner of that row, the single leader from Replication Topologies that everyone agreed would decide the order. You cannot add a "write replica": if two boxes both accepted writes to the same account, you'd be back in the split-brain that all of §3 was spent avoiding. So when the write rate outgrows one machine, no amount of copying rescues you. There is only one move left, and it is a different kind of move: stop copying the data, and start splitting it — so that different writes land on different owners, each responsible for a disjoint slice.

That split is what this section is about, and this lesson is the map of it. Splitting data into pieces is partitioning. Doing it across separate machines is sharding. They are not two words for the same thing, and the difference — which turns out to be the difference between reorganizing your data and actually scaling your writes — is the whole lesson.

In this lesson: the exact relationship between partitioning and sharding (drawn as a 2×2), what splitting within one box does and does not buy you (measured), the one move that actually multiplies your write ceiling (also measured), the bill that move hands you, and the decision most teams get wrong — when to do it at all.

Scope: which key to split on and the hotspots a bad choice creates is Choosing a Shard Key; the ring that remaps gracefully is Consistent Hashing; fixing a shard that melts is Resharding & Hot Shards; splitting by feature instead of by key is Federation. Here they are named on the map, not walked.

Partitioning means dividing one logical dataset into pieces called partitions. That leaves two independent choices, and drawing them as axes makes the whole vocabulary fall into place.

Axis one — direction. Which way do you cut?

• Horizontal: split the ROWS. Same columns, same schema, fewer rows each — users A–M on one piece, N–Z on another. This is the one you'll almost always mean.
• Vertical: split the COLUMNS. Same rows, fewer columns each — the hot profile fields in one table, the rarely-touched blob in another. (Splitting by whole feature — users here, billing there — is the functional flavor, and it earns its own lesson, Federation.)

Axis two — distribution. Where do the pieces live?

• Within one box: all partitions on a single server (Postgres calls this declarative partitioning).
• Across boxes: partitions on different servers. This — and only this — is sharding.

Put the axes together and you get the single most useful sentence in the topic: sharding is horizontal partitioning across machines. Which gives you the relationship people constantly muddle — every shard is a partition, but not every partition is a shard. A partition becomes a shard the moment it lives on its own machine.

Three of the four cells just reorganize data on hardware you already own. Only one cell — horizontal × across-boxes — adds a machine, and therefore adds write budget. Everything else in this lesson is a consequence of that one fact. Think of it with the oldest analogy in the book: partitioning is organizing one library into sections; sharding is opening branch libraries across the city. Same building versus more buildings.

Splitting a table within a single server is a genuinely good tool — it's just not the tool for the job people usually reach for it. Postgres gives you three schemes, and they're worth knowing by name because they'll reappear as sharding schemes too:

• Range — contiguous bands: January's rows here, February's there. Great for time-series and range scans; prone to hotspots (today's partition takes all the writes).
• Hash — hash(key) % N scatters rows evenly across partitions. Kills hotspots; also kills range scans (adjacent keys land far apart).
• List — you name which values go where (this region's rows here).

What within-box partitioning actually buys you is real: the planner can prune — skip partitions that can't match a query — so reads that filter on the partition key touch less data. Dropping a whole month of old data becomes an instant DROP TABLE of one partition instead of a grinding bulk DELETE. Cold partitions can live on cheaper disks. These are maintenance and read-shaped wins, and they matter.

Here is what it does not buy you, and it's the crux of the lesson. We measured it on a real postgres:16 box (capped so the ceiling is stable): a plain table sustained 4,809 writes/second; the same table hash-partitioned eight ways on the same box sustained 4,975 — statistically identical. Of course it was. Partitioning split the table, not the machine. It's the same CPU, the same disk, the same lock manager — the same write ceiling. (Postgres even tells you this in its own guts: partition pruning speeds up reads, but it isn't implemented for the ModifyTable node — the one that runs your INSERTs and UPDATEs.) Within-box partitioning is a manageability move that leaves your write ceiling exactly where it found it.

Now put the partitions on different machines. That's sharding, and it's the one move that changes the number that matters. Same experiment, but the eight partitions become two boxes, each a shard: run write load at both at once and they sustain 4,965 and 4,965 writes/second — 9,930 combined, a clean 2.06× with zero contention, because the two boxes are separate owners with separate budgets. Add a third and it's ~3×; a tenth, ~10×. Sharding is the only cell of the 2×2 that multiplies write throughput, because it's the only one that multiplies machines.

The bar chart is the whole taxonomy in three columns: partitioning within a box (4,809 → 4,975, a flat line) versus sharding across boxes (4,809 → 9,930, a step per machine). One reorganizes; the other scales.

In practice a shard is logical, not a physical server — and keeping those two ideas separate is what makes sharding survivable. Notion, when their Postgres monolith started waking engineers up with write-driven CPU spikes in 2020, sharded into 480 logical shards spread over 32 physical databases — 480 chosen precisely because it divides evenly by so many numbers that they can grow 32 → 40 → 48 hosts and just slide shards over, no re-hashing. Instagram did the same trick years earlier with thousands of logical shards (Postgres schemas) on far fewer servers. You over-provision cheap logical shards up front so that scaling later is moving slices, not re-cutting them. (You won't hand-roll the routing in 2026 — tools like Vitess and Citus present many shards as one database — but they automate exactly the concepts on this map.)

Take a rising write rate and try to keep up. Drag the demand up, then reach for each lever in turn and watch the ceiling bar. A bigger box moves it once, then walls. Read replicas — the reflex from §4 — don't move it at all; writes still funnel to the one leader. Partitioning the table within the box leaves it flat (same machine). Only sharding across boxes multiplies it, and you can dial how many shards. Then, once you're sharded, send a few reads and watch the bill arrive — the case the whole next lesson is about is hiding in one of those buttons.

You bought write budget. Here's the invoice, and it comes due on every operation that has to touch more than one shard.

Cross-shard reads scatter. A query that isn't scoped to one shard's key has to fan out to every shard and merge the results — scatter-gather. And the cost isn't the average shard's latency, it's the slowest one: you can't return until the last shard answers. We measured a read across two shards at 146 ms against a single-shard read's 109 ms — and the gap widens as you add shards, because the maximum of many samples drifts past any single one (sixteen shards each at a 10 ms p99 give a scatter p99 around 20–25 ms). It's the same tail-latency law you met in Quorums ("you finish at the k-th fastest") and PACELC ("you wait for your worst replica"), now at the data layer. The discipline: push the filter down to each shard, never haul rows into app memory to sort them.

Cross-shard joins move data. If two tables are sharded on different keys, joining them means shipping rows between machines. The escapes are to co-locate related data on the same shard (so the join stays local) or to replicate small reference tables to every shard — both of which are really decisions about the shard key.

Cross-shard transactions need a distributed commit. One logical change that touches two shards can't be a normal local transaction — it needs the coordination protocol you'll meet in Two-Phase Commit, which buys atomicity with latency and a new set of failure modes. The entire art of the shard key, then, is to keep your common writes landing on a single shard, so this bill rarely comes due.

And two more line items this lesson only names: the shard key itself is a decision you can't easily reverse (change it and you re-cut every row) — that's Choosing a Shard Key — and adding a shard naively re-homes almost everything, which is what Consistent Hashing and Resharding & Hot Shards exist to soften.

Given all that, the most valuable thing this lesson can leave you with is a bias against doing it. Sharding is a one-way door with a permanent tax, and the strongest voices in the field are blunt about it — the common advice is to avoid distributed approaches "at all costs, until you have less than a year of breathing room" on one tuned box. And one tuned box goes remarkably far: modern hardware scales a single Postgres into the multi-terabyte range, and everything in Database Internals and Scaling Reads buys you years before the write side is genuinely the wall.

So run the diagnosis before you reach for the saw:

• Reads are the bottleneck? That's §4, not §5. Index, replicate, cache — don't shard.
• Storage is the bottleneck? Buy a bigger disk, or tier cold data with within-box partitioning (drop-old-partition). Sharding is a throughput fix, not a capacity fix.
• One hot tenant or key? That's a shard-key problem (Choosing a Shard Key), not a reason to shard everything.
• Haven't exhausted one well-tuned box? Do that first.

Shard when — and only when — your write rate genuinely exceeds the biggest single machine you can buy, and even then, count the cross-shard operations you're signing up for before you commit. The companies you admire for their sharding (Notion, Instagram, everyone running Vitess) all lived on one box for years first, and sharded on a specific, measured write-throughput cliff — not because sharding is what serious engineers do.

Next — Choosing a Shard Key: the single decision that determines whether your shards share the load evenly or one of them melts while the rest sit idle.

The claim the whole lesson turns on — partitioning within a box does nothing for writes, only another box does — is something you can measure in about three minutes. You need docker and pgbench (it ships in the postgres image). The trick is to cap each box at half a CPU so the write ceiling is low, stable, and comparable, and turn off disk flushing so you're measuring the CPU/lock ceiling, not your laptop's SSD.

1. Three boxes, each capped:

for n in plain part shard; do
  docker run -d --name pg_$n --cpus=0.5 -e POSTGRES_PASSWORD=pw \
    -p 5${n:0:1}61:5432 postgres:16 -c fsync=off -c synchronous_commit=off
done   # (use distinct ports; see the notes for the exact three)

2. Give them identical work — one plain table, and the SAME table hash-partitioned 8 ways:

-- plain box:
CREATE TABLE events(id bigserial, user_id int, payload text);

-- partitioned box (SAME machine, 8 partitions):
CREATE TABLE events(id bigint, user_id int, payload text) PARTITION BY HASH (user_id);
SELECT format('CREATE TABLE events_p%s PARTITION OF events FOR VALUES WITH (MODULUS 8, REMAINDER %s)', i, i)
  FROM generate_series(0,7) i \gexec

3. Hammer each with the same write-only load (16 clients, 12s, INSERT a 200-byte row):

echo '\set u random(1,1000000)\nINSERT INTO events(user_id,payload) VALUES (:u, repeat('\''x'\'',200));' > w.sql
pgbench -h localhost -p <port> -U postgres -c 16 -j 4 -T 12 -n -f w.sql postgres

Predict before you run: the plain box hit ~4,800 writes/second. What will the partitioned box do? Here's what we measured:

PLAIN, 1 box ..................... 4,809 tps
PARTITIONED, 1 box, 8 hash parts . 4,975 tps   <- SAME. it's the same machine.

Now the real move — a second box as a second shard, and run write load on both at the same time (background one, foreground the other):

SHARD A .......... 4,972 tps ┐
SHARD B .......... 4,958 tps ┘  = 9,930 tps combined (2.06×), zero contention

That's the entire lesson in two experiments: splitting the table left the ceiling flat; splitting across a second machine doubled it. Clean up: docker rm -f pg_plain pg_part pg_shard.

• "Partitioning and sharding are the same thing." Every shard is a partition; not every partition is a shard. Sharding is partitioning across machines — the word that matters is where the pieces live.
• "Partitioning my table will scale my writes." Within one box it won't — same CPU, same disk, same ceiling (measured: 4,809 → 4,975 tps, noise). It buys pruning, fast drop-old, and cheaper storage tiering. Write scaling needs a second box.
• "We're growing fast, we should shard now." The expert consensus is the opposite: don't, until you have under a year of runway on a tuned box. Sharding is a one-way door with a permanent operational tax.
• "Add a write replica to scale writes." There's no such thing. A datum has exactly one owner (the single leader of §3); replicas serve reads. More write budget means more owners, which means splitting the data.
• "A shard is a server." A shard is logical. Notion ran 480 logical shards on 32 physical databases; Instagram thousands of schema-shards on far fewer servers. You over-provision logical shards so you can move them across boxes without re-hashing every row.
• "Once sharded, queries work the same." Single-shard queries do. Anything spanning shards becomes scatter-gather (you wait for your slowest shard), a cross-shard join (data movement), or a two-phase-commit transaction. The shard key's whole job is keeping common operations single-shard.
• "Sharding is a storage-capacity fix." It's a write-throughput fix. If storage is the problem, buy a disk or tier cold data. Sharding for capacity pays the cross-shard tax for nothing.

• A write has exactly one owner and no copy can absorb it — so §4's read levers can't scale writes. The only move is to split the data into disjoint slices with different owners.
• Partitioning splits a dataset into pieces; sharding puts the pieces on different machines. Every shard is a partition; not every partition is a shard. The taxonomy is a 2×2: direction (horizontal rows / vertical columns) × distribution (within-box / across-boxes), and sharding is the horizontal-across-boxes cell.
• Only sharding multiplies the write ceiling. Measured: one box 4,809 tps; the same box partitioned 8 ways 4,975 (unchanged — same machine); two boxes as two shards 9,930 (2×). Read replicas and within-box partitioning do nothing for writes; a bigger box buys one step, then walls.
• A shard is logical, decoupled from physical servers (Notion 480/32; Instagram thousands/few) — over-provision logical shards so scaling is moving slices, not re-cutting them.
• Sharding bills you every cross-shard operation: scatter-gather reads (you finish at your slowest shard — 146 vs 109 ms measured), cross-shard joins (data movement), cross-shard transactions (two-phase commit). The shard key's job is keeping the common path single-shard.
• Shard last. Only when the write rate beats the biggest box you can buy — and count the cross-shard tax first. Next — Choosing a Shard Key: the decision that makes or melts your shards.