Resharding & Hot Shards

Two lessons ago, in Choosing a Shard Key, a celebrity melted a shard even though you'd picked a high-cardinality key — because one hot value is a property of your data, not your key. Then in Consistent Hashing you learned that the ring balances the keyspace beautifully and still routes every one of that hot key's requests to the single node that owns it. Both lessons ended on the same line: that's Resharding & Hot Shards. Here it is. The shard is on fire, and it's your turn to put it out.

And the first thing to do is the thing everyone skips: diagnose before you cut. Because the reflex — "a shard is overloaded, add more shards" — is the wrong fix for the single most common cause of a hot shard, and reaching for it wastes the outage while the fire keeps burning.

Think of it as a supermarket checkout. One lane is swamped, the line out the door. There are exactly three reasons that can happen, and each has a different cure. Maybe there are just too many shoppers — open more lanes. Maybe everyone's crowding that one lane because it's the only one that sells lottery tickets — opening lanes won't help; the lottery crowd still funnels to lottery. Or maybe one wholesale buyer with three overflowing carts is jamming everyone — give them their own register. Too much volume, one hot item, one whale. Split, salt, isolate.

In this lesson: how to tell which of the three you've got, the three fixes and exactly when each works (and, for the reflex, exactly when it fails), what each one costs, and how you actually reshard a live system without taking it down.

Scope: which key to shard on is Choosing a Shard Key; the ring and virtual nodes are Consistent Hashing — a split just moves tokens on it. This lesson is what you do when, despite both, a shard melts anyway.

Every hot-shard fix is a bet on the cause, so name the cause first. Pull the telemetry that matters — not just requests-per-shard, but requests-per-key — and one of three pictures emerges.

Volume. The shard's load is spread across many keys; it's just too much for one box. On a requests-per-key view, no single key dominates — the shard is uniformly busy. This is the honest "we outgrew a shard" case, and it's rarer than people think under hash sharding (hashing spreads keys evenly by construction), but it happens when a shard's range accumulates a fat slice of the data.

One hot key. A single key is a huge fraction of the shard's traffic — a viral post's like counter, a trending hashtag, a celebrity's profile. On the per-key view it's unmistakable: one bar towering over thousands of flat ones. This is the celebrity from Choosing a Shard Key, arrived.

A whale tenant. One customer or account generates far more than the rest — an enterprise tenant with a thousand times a normal account's activity. Their many keys, taken together, dominate whichever shards hold them; no single key is hot, but one owner is.

These aren't academic distinctions — they map one-to-one to the three fixes, and applying the wrong one is the difference between a five-minute fix and a wasted afternoon while the pager keeps going. So: volume, key, or tenant?

Split is the obvious move: divide the hot shard's key-range into two (or more) shards, so each takes a slice of the old load. When the shard is hot from volume, this is exactly right. We measured a shard owning a fat 40% of the keyspace; split its range and each half takes about half — 1.60× the average down to 1.00×, dead even. Open more lanes, the crowd spreads.

Now the trap, and it's the most important thing in this lesson. Reach for split when the shard is hot from a single key, and it does nothing — worse than nothing. Here's the measured proof. One key is 80% of the writes. Split to 4 shards: the hot key hashes to one of them, which now holds 85% of all traffic. Split to 8: still one shard, 82%. Split to 16: still one shard, 81%. The peak-to-average went 3.4× → 6.6× → 12.96× — you added twelve shards and made the imbalance nearly four times worse, because splitting only divided the cold keys, dropping the average while the hot shard stood exactly where it was.

This is Consistent Hashing's lesson made operational: the ring — and a split — balances the keyspace, not per-key load. A single key hashes to exactly one place, and no amount of splitting changes that. You cannot place your way out of one hot key. It's a property of your data, and you fix it by changing the key, not the shards — which is the next section.

When split is the answer (volume), the hard part isn't deciding — it's doing it while the system serves live traffic. You don't take an outage to fix an overload. The production procedure, which Vitess turns into one command precisely because doing it by hand is where clusters die: copy the rows to the new shards, keep them current by streaming ongoing writes to both (a temporary period of doubled write load), verify the copies match with a row-level diff, then cut over traffic atomically and retire the old shard. The old shard serves every read and write until the instant of cutover. MongoDB 5.0 made this an online operation too; before it, resharding meant the dump-and-reload horror you saw priced in Choosing a Shard Key. If your resharding plan needs downtime, the plan isn't finished.

If you can't place your way out of one hot key, you split the key instead. Salting appends a bounded suffix to the hot key — likes:post42 becomes likes:post42:0, …:1, up through …:N-1 — so what was one scorching key becomes N cool ones, scattered across N shards. Each write picks a random sub-key; the load that was crushing one shard now spreads across N. Instagram's canonical version: one like-counter became many sub-counters, each incoming like landing on a random one.

It works, measurably. The same hot key that made split's peak 12.96× drops toward even as you add sub-keys, and the throughput math is clean and linear: N sub-keys buys N× the write capacity. Ten sub-keys turns a counter that maxed out around a thousand writes a second — a single partition's ceiling — into ten thousand. Google ships this as a first-class primitive (Firestore's distributed counters, Bigtable key salting) because it's the standard answer.

But be honest about two things, because salting is oversold. First, it doesn't eliminate the hotspot — it divides it. One hotspot of strength one becomes N hotspots of strength 1/N; each is now survivable, but the heat didn't vanish, it split. Second, and this is the real bill: reads pay for it. To read the true count you must now query all N sub-keys and sum them — one logical read became N backend reads, multiplying read load, and you don't have your answer until the slowest of the N responds. That's the tail-latency law you've met all course — Quorums' "finish at the k-th fastest," PACELC's "wait for your worst," the scatter-gather of Partitioning vs Sharding — arriving one more time. Salt trades a write hotspot for a read fan-out. For a write-heavy hot key (a counter), that trade is almost always worth it; for a read-heavy one, it's the wrong trade — you'd cache it (Caches & Databases) instead.

And the count of sub-keys is a knob you can get wrong both ways: too few and the sub-keys are still hot; too many and every read fans out further for no benefit. There's a sweet spot, and it moves with the write rate.

The third case isn't one hot key — it's one hot owner. A whale tenant whose thousands of keys, together, keep melting whatever shards they land on. You can't salt them (there's no single key to split), and splitting just smears their noise across more of your shared shards — now the noisy neighbour is everywhere instead of contained.

The fix is the pragmatic one: isolate. Pull the whale onto its own dedicated shard (or replica set), and put a per-tenant quota in front so no single tenant can ever consume the whole cluster again. The shared fleet is instantly relieved — everyone else's shards go back to even — and the whale gets its own capacity budget, which you can size for them specifically. Its dedicated shard runs hot, and that's fine: it's theirs, and it hurts no one else. AWS does exactly this automatically — DynamoDB On-Demand isolates high-traffic partitions onto their own capacity — and it's a standard move for enterprise SaaS, where one Fortune-500 customer legitimately dwarfs a thousand small ones.

Isolation is often better than a heavy reshard precisely because it's targeted and reversible: you move one tenant, not the whole keyspace, and you can move them back. Add the per-tenant quota and you've also turned an emergency into a policy — the next whale gets throttled at their limit instead of taking down the shard.

A shard is melting. Diagnose why — too much volume, one viral key, or a whale tenant — then pick your fix and watch it work. Try the reflex first: on the viral-key ticket, hit split and add shards, and watch the peak get worse while one shard stays at 80%. Then salt the key and watch the bars level out — with the read fan-out that's the price of it. Each cause has exactly one right answer; the widget grades every choice and tells you why the wrong ones are wrong.

The claim that adding shards can't fix a hot key is the kind of thing you should see with your own eyes, because it's so counterintuitive. Here it is in ~15 lines of Python with real hashing — no dependencies.

import hashlib, random
random.seed(1)
h = lambda s: int(hashlib.md5(s.encode()).hexdigest(), 16)

# one key, 'post42', is 80% of all writes; the rest are spread over 500 cold keys
cold = [f'post-{i}' for i in range(500)]
stream = ['post42' if random.random() < 0.80 else random.choice(cold) for _ in range(200_000)]

def hottest_share(n_shards):
    load = [0] * n_shards
    for k in stream: load[h(k) % n_shards] += 1
    return max(load) / sum(load) * 100      # % on the busiest shard

for n in (4, 8, 16, 32):
    print(f'{n:>3} shards -> hottest shard has {hottest_share(n):.0f}% of all writes')

Predict first: as you go from 4 shards to 32, does the busiest shard's share go down? Then run it:

  4 shards -> hottest shard has 85% of all writes
  8 shards -> hottest shard has 83% of all writes
 16 shards -> hottest shard has 81% of all writes
 32 shards -> hottest shard has 81% of all writes

Eight times the shards and the fire didn't move — the one hot key hashes to one shard, and the other 31 sit nearly idle. (The peak-to-average actually rises, because the mean drops while the hot shard holds steady.) Now salt the key and watch it finally spread — each write picks a random sub-key:

def salt(sub_keys, n_shards=16):
    load = [0] * n_shards
    for k in stream:
        key = f'post42:{random.randint(0, sub_keys-1)}' if k == 'post42' else k
        load[h(key) % n_shards] += 1
    return max(load) / sum(load) * 100

for s in (1, 5, 10, 20):
    print(f'{s:>2} sub-keys -> hottest {salt(s):.0f}%')   # 1 -> 81% ... 20 -> ~14%

Salting the key spreads what splitting the shards couldn't. That's the whole lesson in two loops.

Put it together as a decision, because the whole lesson is that the fix follows the cause:

• Hot from volume (many keys, no single one dominant) → split the shard, live.
• Hot from one key (a viral counter, a trend) → salt the key; accept the read fan-out. Splitting shards will not help — that's the trap.
• Hot from one tenant (a whale) → isolate them onto a dedicated shard with a quota.

And the most senior move of all — knowing when not to reach for any of these:

• A write-once key is never a write hotspot. A user's profile is written at signup and never again; salting user:id buys nothing but read cost. Don't fix a fire that isn't burning.
• An already-even key needs nothing. user_id across millions of users is already spread by hashing (Choosing a Shard Key's point: the cardinality was fine; it was the one hot value that burned). Salt the hot value, not the whole key.
• A read-hot key wants a cache, not a salt. Salting is a write-distribution tool. If the key is hot on reads, the answer is Caches & Databases — put it behind a cache or a read replica — not a fan-out counter that makes reads more expensive.

What you'd say under follow-up: "First I'd look at requests-per-key, not just per-shard, to find the cause. Volume → split, live, with double-writes and a verified cutover. One hot key → salt it into N sub-keys — writes scale linearly, reads fan out and I'd tune N to the write rate — and I would not add shards, because a single key hashes to one place no matter how many. A whale → isolate onto a dedicated shard with a per-tenant quota. And if it's a read-hot key or a write-once key, none of these — cache it, or leave it alone."

• "Shard hot? Add more shards." Only works for volume. For one hot key, splitting is useless — the key still hashes to one shard (measured: 85%/82%/81% across 4/8/16 shards), and the peak-to-average gets worse (3.4× → 13×). You can't place your way out of one hot key.
• "Salting eliminates the hotspot." It divides it — one hotspot of strength 1 becomes N of strength 1/N. Each is survivable; the heat split, it didn't vanish.
• "Salting is free." The bill is a read that fans out to all N sub-keys and sums, multiplying read load and waiting for the slowest sub-shard. Write-cheaper, read-costlier — worth it for a write-hot counter, wrong for a read-hot key.
• "Salt anything hot." Only write-hot keys. A write-once key (a profile) is never a write hotspot; an already-even key is already spread; a read-hot key wants a cache (Caches & Databases).
• "Resharding is a config change." It's an online data migration — copy, double-write to keep current, verify, cut over — with temporarily doubled write load. If it needs downtime, the design is incomplete, and by-hand is where clusters die (which is why Vitess and MongoDB made it a workflow).
• "Consistent hashing already prevents hot shards." It balances the keyspace, not per-key load. A single hot key defeats any hash. This lesson is the fix the ring can't provide.
• "Isolating a tenant is a hack." Dedicating a shard to a whale plus a per-tenant quota is a legitimate, targeted, reversible production choice — often better than a heavy reshard.

• When a shard melts, diagnose the cause before you cut — the fix follows the cause: volume → split, one hot key → salt, a whale tenant → isolate.
• The reflex (split) fails on the most dangerous cause. A single hot key hashes to one shard no matter how many you add (measured 85%/82%/81% at 4/8/16 shards; peak 3.4× → 12.96×, worse). You cannot place your way out of one hot key — that's Consistent Hashing's "keyspace, not per-key load," operational.
• Salt splits the key into N sub-keys → N× write throughput, linearly (10 → 10,000 writes/s, like DynamoDB). It divides the hotspot into N of strength 1/N; the bill is a read fan-out to all N sub-keys, paying the tail-latency tax. Only for write-hot keys.
• Isolate a whale tenant onto a dedicated shard with a per-tenant quota — relieves the shared fleet, targeted and reversible, often better than a heavy reshard.
• Resharding is a live operation, not a config change: copy → keep-current (doubled writes) → verify → cut over, no downtime. And the senior move is knowing when to do nothing — write-once keys, already-even keys, and read-hot keys (cache those). Next — Federation: split by function before you split by key.