Batching & Continuous Batching

Introduction

Two lessons ago (L207) we found the central tension of serving: latency is per-user, throughput is the bill, and they trade off. Last lesson (L208) we found the budget that gates it all — the KV cache. This lesson is the lever that turns that budget into throughput: batching. It's the single biggest GPU-utilization unlock in LLM serving, and the modern form — continuous batching — is why an open-source server like vLLM gets ~24× the throughput of a naive setup on the same hardware.

A GPU serving one user at a time is almost entirely idle — around 5% utilized. Not because it's slow, but because (from L208) decode re-reads the entire model from memory for every single token. Batching is how you make that enormous read pay for many users at once.

In this lesson:

Why batching is near-free throughput — the memory wall, and how a batch amortizes the weight read
Static (naive) batching and its fatal flaw — head-of-line blocking (waiting for the slowest request)
Continuous (iteration-level) batching — the Orca/vLLM idea that keeps the GPU full
The catch — why attention doesn't batch like the rest of the model
The dial — bigger batches buy throughput at the cost of per-user latency, and how to schedule around it

Scope: this is batching as a scheduling idea. It rests on the KV cache + PagedAttention from L208 (the memory that makes swapping requests in and out feasible) — we use that here, we don't re-derive it. Speculative decoding (a different throughput trick) is L210; the actual serving frameworks that implement all of this (vLLM / TGI) are L211. We'll name them, and go deep there.

Infographic titled 'Batching & Continuous Batching'. The big idea: batching is how you turn the GPU's idle memory bandwidth into throughput, and continuous (iteration-level) batching is the scheduling trick that keeps the GPU full. LEFT, WHY BATCH — the memory wall: to generate each token the GPU must read the ENTIRE model weights from memory (about 140 GB for a 70B model in FP16) to do only a few hundred million floating-point operations, so a single user leaves the GPU around 5 percent utilized; batching 32-64 requests amortizes that one weight read across all of them, raising arithmetic intensity, so throughput soars and cost-per-token drops. CENTER, two schedules of a GPU with 4 slots (rows) over iterations (time). STATIC BATCHING: a fixed group runs together and the batch cannot advance until its SLOWEST request finishes, so short requests finish and their slots sit IDLE (red, wasted) — head-of-line blocking — and new requests must wait for the whole batch to clear; the example runs 14 iterations at only 57 percent utilization. CONTINUOUS BATCHING: the moment a request finishes, its slot is refilled from the queue on the very next iteration, so the slots stay packed; the same work finishes in 10 iterations at 80 percent utilization. THREE CARDS. Card 1, why batch: decode is memory-bandwidth-bound (L207/L208), so amortizing the weight read across a batch is nearly free throughput until the GPU becomes compute-bound. Card 2, continuous = iteration-level scheduling: Orca (2022) reforms the batch every forward pass — finished sequences leave, queued ones join, and prefill and decode are mixed in one batch; combined with PagedAttention KV-cache management (L208) this gave vLLM about 24x the throughput of HuggingFace Transformers and Orca about 37x over FasterTransformer at equal latency. Card 3, the dial: bigger batches raise per-user time-per-output-token, so you cap the batch to hold the latency SLO (maximize goodput, not raw throughput); chunked prefill splits a long prompt into 256-512-token chunks interleaved with decodes to stop a big prefill from stalling everyone's tokens. A catch is also noted: attention does not batch like the rest of the model because each sequence has its own K/V cache, so attention stays per-sequence (selective batching). Section roadmap strip: metrics, KV cache, batching, speculative decoding, serving with vLLM/TGI, streaming. Takeaway banner: batching amortizes the weight read for throughput, continuous batching schedules at the iteration level so the GPU never idles on a finished request, and you size the batch to your latency SLO.

Why Batch at All? — The Memory Wall

Start from the bottleneck we proved in L207–L208: decode is memory-bandwidth-bound. To generate one token, the GPU reads the entire model's weights out of memory — for a 70B model in FP16 that's ~140 GB of data moved to do only a few hundred million FLOPs. The GPU's math units are world-class; they spend almost all their time waiting on memory. Serve a single user and your expensive accelerator sits at roughly 5% utilization.

Here's the trick: that 140 GB weight read is the same for everyone. So if you process many requests' decode steps together — a batch — you pay the giant read once and reuse it across all of them. Load the weights, and in the same pass advance 64 different conversations by one token each.

Batching raises arithmetic intensity — more compute per byte read. The numerator (FLOPs) grows with the batch while the denominator (weight bytes moved) stays fixed. The GPU shifts from memory-starved toward compute-saturated, and throughput climbs almost linearly with batch size — nearly for free — until the GPU finally becomes compute-bound and saturates.

This is the whole reason batching exists: it's not just "do things in parallel," it's amortizing the dominant cost (the memory read) across many users. It's also the L207 dial made real — more batch = more throughput = lower cost per token.

Static Batching & Head-of-Line Blocking

The naive way to batch is static (a.k.a. fixed) batching: collect N requests, run them through the model together, wait for all of them to finish, then start the next batch. For a fixed, uniform workload that's fine. For real LLM traffic it's badly inefficient, for one reason: requests generate wildly different numbers of tokens, and you can't know how many in advance.

That breaks static batching in two ways:

Head-of-line blocking (the big one): the batch can't finish until its slowest member finishes. A request that produces 8 tokens holds the whole batch for 8 iterations — so the requests that finished at token 2 sit in their GPU slots done but idle, burning capacity, unable to leave. One long generation stalls all the short ones sharing its batch.
Queueing + padding: new requests must wait for the entire current batch to clear before they can even start (terrible TTFT under load), and prompts of different lengths get padded to the longest, wasting more compute.

Picture 4 GPU slots running a batch where one request needs 8 tokens and three need 2. After iteration 2, three of four slots are idle for six more iterations — the GPU is ~40% wasted, and nobody new can board. The interactive below makes this painfully visible. Static batching effectively serves at the pace of its slowest request.

Continuous Batching — Schedule at the Iteration Level

The fix, introduced by the Orca paper (OSDI 2022) and popularized by vLLM, is continuous batching — also called iteration-level scheduling or in-flight batching (NVIDIA's name for it). The idea is simple and powerful: don't schedule whole batches — schedule each forward pass.

Every iteration (every token step), the scheduler re-forms the batch:

A request that finished this step leaves immediately, freeing its slot.
A request waiting in the queue joins immediately to fill that slot — even mid-flight, while others keep decoding.
New requests can prefill in the same batch where others are decoding (prefill and decode are mixed).

So the GPU never sits idle waiting for the slowest request — the instant a slot opens, the next request boards. In pseudocode the difference is the whole lesson:

# STATIC batching — wait for the whole batch, every time
while queue:
    batch = queue.take(N)                 # grab N requests
    while any(r.unfinished for r in batch):
        step(batch)                       # everyone runs until the SLOWEST is done
    # short requests sat idle here, waiting; only now can new requests start

# CONTINUOUS batching — re-decide every single iteration
running = []
while queue or running:
    for r in list(running):
        if r.finished:
            running.remove(r); emit(r)    # finished -> leave NOW, free the slot
    while len(running) < N and queue:
        running.append(queue.pop())       # queued -> join NOW, fill the slot
    step(running)                         # one token for everyone currently running

The payoff is enormous because it's pure scheduling — no quality change, same model. This is where the famous numbers come from: Orca reported up to 36.9× throughput over FasterTransformer at the same latency, and vLLM's first release hit 24× over HuggingFace Transformers (and 3.5× over HuggingFace TGI) — the combination of continuous batching (scheduling) and PagedAttention (the L208 KV-cache memory management). They're a pair: iteration-level scheduling decides who runs, and PagedAttention makes it cheap to swap requests' KV caches in and out of GPU memory without fragmentation. Continuous batching is the default in every serious serving stack today (vLLM, TGI, TensorRT-LLM, SGLang).

See It — The Batching Lab

Watch the difference directly. Below, a GPU with 4 slots runs a queue of requests over time. Flip static ↔ continuous, and switch the workload between even and varied request lengths:

A GPU with 4 slots (rows) runs requests over iterations (time) — each block is one request decoding a token per step. Flip STATIC vs CONTINUOUS and switch the workload between even and varied request lengths. With STATIC + varied lengths, short requests finish but their slots sit idle (red ×) until the slowest one in the batch finishes — head-of-line blocking, ~57% utilization. With CONTINUOUS, each finished slot is refilled from the queue the next iteration, so the GPU stays full (~80%) and the same work drains in fewer iterations. Note that even-length traffic hides the whole problem — which is exactly why real, variable traffic needs continuous batching. Illustrative schedule.

What to notice:

Even lengths hide the problem — static and continuous tie (everyone finishes together, no idle slots). This is why a benchmark with uniform outputs makes static batching look fine.
Varied lengths (real traffic) expose it: static strands short requests in idle slots (the red ×) waiting for the 8-token request — 57% utilization, 14 iterations. Continuous refills each slot the moment it frees — 80% utilization, 10 iterations, same work. That gap, multiplied across a busy server, is the 24×.
Notice continuous still has a small idle tail (the last long request drains alone). It's not magic — it just never wastes a slot it could have filled.

The Catch — Attention Doesn't Batch

One honest wrinkle, and it connects straight back to L208. Batching makes the model's big feed-forward (MLP) layers efficient: every request multiplies by the same weight matrix, so they combine into one fat matrix-matrix multiply that lights up the GPU's tensor cores. But attention is different — every sequence has its own KV cache (its own keys and values), so there's no shared matrix to batch against. Attention stays a pile of independent, per-sequence matrix-vector products — exactly the memory-bound pattern from L208.

This is why Orca also introduced selective batching: batch the parts that share weights (the FFN), but run attention per-sequence. And it's why batching isn't infinite free lunch: as you push batch size and context length up, the per-sequence attention work grows until it — not the amortized weights — becomes the bottleneck, and throughput gains flatten.

The shape of the whole section rhymes: the KV cache (L208) is what makes attention cheap enough and swappable enough for continuous batching to work — and it's also the thing that eventually caps how far batching can scale. The cache is always the budget.

The Dial — Bigger Batches, Slower Tokens

Batching is not a free "set it to 256" win. It's the throughput-vs-latency dial from L207, and turning it up has a cost: bigger batches raise every user's TPOT. More sequences share each forward pass, the per-step attention work grows, and each user's tokens arrive a little slower. Push the batch too high and you blow your TPOT SLO — so the real objective isn't max throughput, it's maximum goodput: the most requests/second that still meet the latency target (recall L207). You size the batch to the SLO, then stop.

There's a subtler scheduling problem too: prefill–decode interference. A new request's prefill is compute-heavy (it processes the whole prompt at once); if you drop it into a batch that's busy decoding, that one fat prefill bottlenecks the step and every in-flight user's next token stalls (a TPOT spike). The common fix is chunked prefill (from Sarathi-Serve): split a long prefill into 256–512-token chunks and interleave them with ongoing decodes, so no single step is dominated by prefill and TPOT stays smooth. vLLM and TensorRT-LLM both ship adaptive versions of this.

The takeaway for you as an app engineer: you rarely write a scheduler, but you choose the knobs — max batch size, max sequence length, and whether to enable chunked prefill — and those choices are exactly the latency-vs-throughput-vs-cost trade from L207. (The phase-separation extreme — running prefill and decode on different machines, DistServe-style — is an advanced serving topic for L211.)

🧪 Try It Yourself

Reason through these, then confirm with the lab and the calculator:

Predict: in the lab with varied lengths, why does static sit at ~57% utilization while continuous hits ~80%?
Your benchmark shows static and continuous batching performing identically. What's almost certainly true about your test workload — and why is it misleading?
A single user sends one request to an idle server. Will continuous batching make their response faster? Why or why not?
You raise max batch size from 16 to 128 and throughput barely improves while users complain it got slower. Give two reasons (one from "the dial," one from "the catch").
Long documents arriving for summarization keep causing TPOT spikes for everyone else mid-stream. What scheduling feature fixes this, and how?

→ (1) Head-of-line blocking: static must wait for the slowest request in each fixed batch, so short requests leave idle slots (~43% wasted); continuous refills each freed slot the next iteration, keeping the GPU full. (2) Your requests are all ~the same length — even-length traffic hides head-of-line blocking entirely. Real traffic has high length variance, where continuous wins big; a uniform benchmark over-flatters static. (3) No — batching raises throughput, not single-request latency. With one user there's nothing to batch with; their TTFT/TPOT are unchanged (if anything, a busy batch would slow them). (4) Dial: a bigger batch raises per-user TPOT (slower tokens) and throughput saturates once you're compute-bound; Catch: attention doesn't batch (each sequence's KV is separate), so past a point attention dominates and adding batch stops helping. (5) Chunked prefill — split the long prompt into 256–512-token chunks interleaved with the ongoing decodes, so the big prefill no longer monopolizes a step and everyone's TPOT stays smooth.

Mental-Model Corrections

"Batching just means processing requests in groups." It means amortizing the weight memory read across requests — a memory-bandwidth trick (decode is memory-bound), not mere parallelism.
"Continuous batching means bigger batches." No — it's iteration-level scheduling (swap finished out / queued in every step). The win is keeping the GPU full, independent of batch size.
"Bigger batch is always better." Throughput saturates, and per-user TPOT rises. Size the batch to your SLO (goodput), not to the maximum.
"Batching makes responses faster." It raises throughput and lowers cost/token, usually at slightly worse per-user latency. It does not speed up a lone request.
"Static batching is fine." Only for uniform-length workloads. Real traffic has high length variance, where head-of-line blocking wastes a huge fraction of the GPU.
"All parts of the model batch the same." No — the FFN batches (shared weights → matrix-matrix), but attention is per-sequence (each request's own KV cache, L208), so it stays the bottleneck at scale.
"Throughput is the goal." Goodput is — throughput that meets the latency SLO. That's what caps your batch size.

Key Takeaways

Batching = amortizing the weight read. Decode re-reads the whole model per token (memory-bound, L207/L208), so one user wastes ~95% of the GPU. A batch pays that read once for many users → throughput up, cost/token down — nearly free until compute-bound.
Static batching has a fatal flaw: head-of-line blocking. It waits for the slowest request, stranding short ones in idle slots and making new requests queue behind the whole batch.
Continuous (iteration-level) batching fixes it. Re-form the batch every forward pass — finished leave, queued join, prefill mixes with decode. Same model, pure scheduling: ~24× (vLLM) / ~37× (Orca) throughput.
It's powered by the KV cache + PagedAttention (L208) — cheap swapping of request caches is what makes iteration-level scheduling feasible.
The catch: attention doesn't batch (each sequence's KV is separate → selective batching), so gains flatten as batch and context grow.
The dial: bigger batch = more throughput but higher TPOT. Size the batch to your goodput/SLO; use chunked prefill to stop big prefills from stalling everyone's tokens.
Next — L210: Speculative Decoding & Other Tricks — batching attacks throughput by sharing the weight read; speculative decoding attacks latency by generating several tokens per forward pass. Two different levers on the same memory wall.