Write Batching & Buffering

Watch a single small write land in a database and time where the milliseconds actually go. Almost none of it is the write. The bytes are tiny; copying them is nearly free. The cost is everything around the write: a network round trip to the server, parsing and planning the statement, appending to the write-ahead log, and — the big one — an fsync to force the commit to disk so it survives a crash. Call all of that the toll: a fixed cost you pay per operation, no matter how small the write. On top of the toll sits a small variable cost that scales with the actual bytes. For a big write the bytes dominate and the toll is noise. For a small write — a log line, a metric, a click event, a row — the toll is nearly the whole bill.

That reframes the write-scaling problem. All of §5 so far scaled writes by splitting the data across more boxes — sharding and federation (Partitioning vs Sharding through Federation) — which adds owners so more writes can happen in parallel. This lesson is the other axis: don't add boxes, make each write cheaper on the box you already have. And the way you make a small write cheaper is almost insultingly simple: stop paying the toll on every single one. Pay it once, for a whole group. That's batching, and the two axes compose — shard across boxes, batch within each.

Picture a delivery truck. A truck costs the same to send whether it carries one box or a hundred — the driver, the fuel, the drive to the depot and back are a fixed cost per trip. Ship boxes one at a time and every box pays for a whole truck. Wait until the truck is full and a hundred boxes share one trip — the cost per box falls off a cliff. Everything else in this lesson — why it's not free, when it backfires — falls out of that one picture.

Put numbers on it. Writing N rows one at a time costs N × (toll + work). Batched into one operation, it costs toll + N × work — you paid the toll once. So the cost per write falls from toll + work toward the bare work floor, shrinking as toll / N. That's a 1/N curve, and its most important property is where the savings live: the fraction of the toll you kill is 1 − 1/N, which means most of the win is in the first few writes. N=2 kills half the toll. N=4 kills three-quarters. N=10 kills ninety percent. You do not need giant batches — you need to stop paying the toll every single time.

We measured it on Postgres, inserting the same 20,000 small rows in batches of increasing size (each batch = one round trip, one commit). The speedup versus one-row-at-a-time:

Read that curve carefully, because it has three regions and every one is a lesson. First, the steep climb: the measured speedup tracks the 1 − 1/N law almost exactly (2× at 2, 3.8× at 4, 7.3× at 8, 14.3× at 16) — the cost per row collapses from 644 µs to about 6 µs. Second, the knee: by a batch of 64 you're already at 43×, having killed 98% of the toll; the gains flatten because there's barely any toll left to amortize. Third — and this is the part tutorials skip — the regression: the peak is at batch 1,000 (109×), and batch 5,000 drops back to 94.9×. Batches that are too big start to hurt: a giant request eats memory, takes one long parse, and can hold locks longer. The curve goes up, flattens, then dips. The whole engineering instruction is one sentence: tune to the knee, not past it — a few hundred rows, not a million.

Once you see "pay the fixed toll once," you start seeing it in every layer of the stack — the same idea aimed at different tolls. This is worth collecting, because interviewers love to hear you name the pattern, not just one instance of it.

• Group commit aims it at the fsync. One fsync per commit is safe but slow. If ten transactions commit at nearly the same moment, the database flushes the log once and acknowledges all ten — the disk-sync toll is paid once, not ten times. The elegant part: group commit largely self-clocks with concurrency — the more clients committing together, the bigger the natural batch, no tuning required. This is WAL & Durability's promise cashed: the same fsync that heals a crash also batches your commits.
• Redis pipelining aims it at the network round trip — and it's the cleanest number in the lesson. A Redis command is ~0.1 ms of actual work but 1–2 ms of round trip; the network is 10–20× the work. Send 1,000 commands one at a time and you pay 1,000 round trips. Pipeline them — fire all 1,000, then read all the replies — and you pay one. We measured 1,000 SETs go from 113 ms to 9.9 ms on localhost (an 11× win with a tiny 0.1 ms loopback); over a real 1–2 ms network the same batch goes from ~1,500 ms to ~10 ms — the classic ~80×. The server did the identical work; you just stopped paying the round-trip toll per command.
• Kafka producers aim it at the request: batch.size (bytes) plus linger.ms (time), flushing on whichever fires first — 10–50× the throughput of sending one message at a time.
• Nagle's algorithm aimed it at the packet back in 1984 (RFC 896): TCP itself holds a tiny send until the previous one is acknowledged, coalescing your small writes into full packets. Batching is so fundamental it's been baked into the network stack for forty years. (Kafka's docs literally describe linger.ms as "analogous to Nagle's algorithm.")

It's all one move. LSM trees (#16) do it too — buffer writes in a memtable, flush them to disk as one sequential run — and so does a write-behind cache. Different toll, same trick.

Nothing is free, and batching's bill is latency. Box number one has to wait in the truck until it fills; a write that lands in an empty buffer waits for the batch to complete before it's done. Bigger batches amortize harder and make each write wait longer. That's the fundamental throughput-for-latency trade, and it forces a design decision: when do you flush?

There are two triggers, and mature systems use both, whichever fires first. Flush by size (the buffer hits N items) bounds your throughput and memory. Flush by time (a linger timer of T milliseconds) bounds your latency. You need the time bound because of a failure mode that bites every size-only batcher: at low traffic the batch never fills. If you flush only at 500 rows and only 3 rows an hour show up, those 3 rows wait an hour. So Kafka pairs batch.size with linger.ms; Postgres has commit_delay; every real buffer has a flush interval. Size for the firehose, time for the trickle — and never ship size-only.

Now the part that trips up even strong engineers, because it sounds backwards: batching does not always add latency — at high load it can reduce it. When writes arrive faster than the system can pay tolls, doing them one at a time makes the toll a bottleneck and the queue backs up — your latency is dominated by waiting in line, and it climbs without bound. Batching drains that queue far faster, so end-to-end latency actually drops despite the linger. This is why Kafka 4.0 changed the default linger.ms from 0 to 5: they found that for most workloads a small linger produces "similar or lower producer latency" than no batching at all. The honest rule isn't "batching adds latency" — it's batching adds latency when you're idle and removes it when you're slammed. (You can feel exactly this in the widget: at a low arrival rate, batching stalls each write; crank the rate up and watch the batched pipeline stay smooth while one-at-a-time chokes.)

There's a second price, quieter and more dangerous than latency: a write sitting in a buffer is not durable. The whole speedup of buffering comes from acknowledging a write before it's safely on disk — which means a crash between the acknowledgement and the flush loses that write, silently. Worse, once a buffered write is accepted, its later failure often can't be reported back to the caller — the original call already returned success. You can be told a write succeeded and lose it anyway. So batching for durability is really a single blunt question: how much data can you afford to lose to go how much faster?

MySQL turns that question into one dial — innodb_flush_log_at_trx_commit — and it's the crispest artifact in the whole lesson:

• = 1 (fsync each commit): flush to disk on every commit. Survives any crash. Slowest. This is zero data loss.
• group commit: still durable, but many commits share one fsync — durable and fast, at the cost of a hair of latency. The sweet spot for most databases.
• = 2 / = 0 (async, buffered): acknowledge from memory, flush about once a second. Much faster — but a crash loses up to a second of "committed" writes.

We measured the two ends on Postgres (whose synchronous_commit is the same idea): fsync-on-every-commit ran at 1,516 rows/sec; flipping to async buffered ran at 7,976 rows/sec — 17.9× faster. That 17.9× is not free performance; it's the exact price of the loss window you just opened. Metrics, logs, click-streams? Open the window — losing a second of telemetry on a rare crash is fine. A payment ledger? Keep it shut, or use group commit so you get most of the speed without the loss.

And latency has its own infamous cautionary tale here. Nagle's algorithm — that helpful packet-batcher — has a dark side: combine it with the receiver's delayed ACK (which waits ~40 ms to piggyback an acknowledgement) and the two well-meaning optimizations deadlock — the sender won't send until it's ACKed, the receiver won't ACK yet — producing a fixed ~40 ms stall on interactive traffic. The fix is TCP_NODELAY: disable batching for latency-sensitive paths (SSH, gaming, RPC, database clients) and keep it only for bulk transfer. Two sensible batching optimizations, combined, became a latency disaster — the price made infamous.

This is a tradeoff you should feel, not memorize, so here's the whole thing as a live dial. Writes stream in at a rate you set, pile into a buffer, and flush as one batch when it fills or the linger timer fires — paying the fixed toll once. Watch the three numbers that matter move together: throughput climbs, cost per write falls along the 1/N curve, and latency rises as the batch grows.

Then go find the edges. Load One-at-a-time and see the toll dominate. Slide up to the knee and watch cost-per-write collapse while the curve flattens. Push past the knee and watch latency grow for no throughput. Drop the traffic to a trickle and watch the buffer starve — the writes just sit there until the linger saves them. Crank the rate past what the flusher can drain and watch the buffer back up (that's where you've outgrown one box and sharding takes over). And when you've got a comfortable buffer sitting there un-flushed, hit CRASH — and watch exactly those writes vanish.

Batching is a lossy compression of overhead: you trade latency, a data-loss window, and real operational complexity for throughput. That trade is fantastic in some places and a mistake in others, and knowing the difference is the senior skill. Don't batch when:

• The variable cost dominates. Batching amortizes the fixed toll. If each write is already large — a 10 MB blob — the round trip is rounding error and batching buys almost nothing. Batching pays off precisely when many small writes each pay a big fixed toll.
• Latency is the SLA. A per-user, interactive request should not sit in a batch window waiting for strangers. That's the Nagle lesson: TCP_NODELAY the click path, batch the background firehose.
• You can't afford the loss window and can't group-commit-durably — then fsync each and pay for it.
• The complexity isn't worth it. Batching adds a flush policy, partial-failure handling (if row 57 of 100 fails, is it all-or-nothing or which 43 applied?), retry idempotency (a re-run billing batch can double-charge), buffer sizing, and a backpressure decision — when the buffer fills, do you block the producer (every item matters: payments) or drop (freshness beats completeness: metrics)? An unbounded buffer just moves the failure to an out-of-memory crash on your next traffic spike. That's a lot of new failure surface for a service that isn't paying a meaningful toll in the first place.

So the rule rhymes with the one from Partitioning vs Sharding: batch last, like you shard last. Reach for it when you can see the fixed toll dominating a stream of small, loss-tolerant writes — and leave it alone when you can't.

• Every small write pays a fixed toll — a round trip, an fsync, a commit — that dwarfs the write itself. Batching's whole job is to pay the toll once for many writes. Sharding (#37–#41) added owners to scale writes across boxes; batching makes each write cheaper on one box. They compose.
• The savings follow a 1/N curve with an early knee — the toll you kill is 1 − 1/N, so the first few writes buy most of the win (measured: 2× at batch 2, 14× at 16, 43× at 64). It peaks and then regresses for batches that are too big (measured 109× at 1,000, 95× at 5,000). Tune to the knee, not past it.
• It's one move aimed at different tolls: group commit batches the fsync, Redis pipelining batches the round trip (~80× over a network), Kafka batches the request, Nagle batches the packet.
• The price is latency and durability. Flush on size OR time (time bounds latency and stops low-traffic starvation). Latency is load-dependent — batching adds it when idle, removes it when slammed. And a buffered write is not durable: a crash loses it (MySQL flush_log 0/1/2; measured async is 17.9× faster than fsync-each — that gap is the loss window you accept).
• Batch last, like you shard last — when small, loss-tolerant writes are visibly toll-bound, not before; never on a latency-critical path (TCP_NODELAY) or when the variable cost already dominates. Next — Codelab: Partition a Dataset: you've now got the whole write-scaling toolkit — shard across boxes, batch within each — so let's go partition a real dataset by hand.