Speculative Decoding & Other Tricks

Introduction

Last lesson ended on a clean split: batching attacks throughput by sharing the giant per-token weight read across many requests (L209). This lesson attacks the other side — latency — with a trick that feels like magic the first time you see it: get several tokens out of a single forward pass. It's called speculative decoding, it's lossless (the output is identical to normal decoding), and it's the default fast-path in every serious serving stack.

The whole section has been circling one fact (L207–L209): decode is memory-bandwidth-bound — each token drags the entire model out of memory. Speculative decoding's insight is that that same pass can check several guessed tokens at once, almost for free — so if you can guess ahead and be right often, you generate many tokens per expensive pass.

In this lesson:

The draft-then-verify loop — guess K tokens cheaply, verify them in one target pass, keep the good prefix
Why it's lossless — rejection sampling guarantees the exact output distribution (it is not an approximation)
Where the draft comes from — a small draft model, self-speculation (Medusa / EAGLE), or zero-cost n-gram
The catch — speedup tracks the acceptance rate, and it's a low-batch trick (it fights batching)
Other tricks on the same memory wall — quantization (shrink the read) and faster kernels (FlashAttention)

Scope: this is the technique. The serving frameworks that implement it (vLLM, TGI, TensorRT-LLM) are L211. It builds directly on the memory-bound picture from L207–L208 and the weight-read framing from L209. (You met quantization formats in fine-tuning — there to fit training in memory; here we use them for inference speed.)

Infographic titled 'Speculative Decoding & Other Tricks'. The big idea: get several tokens out of one expensive forward pass, so generation is faster with IDENTICAL output. CENTER, the draft-then-verify loop. Step 1, a small, cheap DRAFTER (a tiny model or an n-gram lookup) guesses the next K tokens given the text so far, for example 'the cache stores past keys'. Step 2, the big TARGET model VERIFIES all K of them in a SINGLE forward pass (cheap, because decode is memory-bound — the pass moves the whole model from memory either way). Step 3, ACCEPT the matching prefix (green) up to the first wrong guess, REJECT that one (red, discarded) and let the target emit one correct token of its own (amber) — so this pass commits 'accepted + 1' tokens for the price of one. Because acceptance is governed by rejection sampling (accept each draft token with probability min(1, p_target/p_draft)), the output distribution is exactly the target model's — it is LOSSLESS, same quality. A callout shows 1 pass yielding about 3 tokens, roughly a 3x speedup. THREE CARDS. Card 1, where the draft comes from: a separate small draft model (about 1B for a 70B, ~60-80 percent acceptance), Medusa heads, EAGLE / EAGLE-3 (hidden-state head with tree drafting, ~70-80 percent, 3-4x), or zero-training n-gram / prompt-lookup that copies tokens from the prompt — superb for code, RAG and summarization. Card 2, the catch: speedup tracks the acceptance rate (aim above 0.6 at K=5; below 0.4 lower K), and it helps at LOW batch where you are memory-bound — at high batch (over ~16) you are already compute-bound, so you skip speculation. Card 3, other tricks that attack the same memory wall: quantization shrinks the weight read (INT8 / FP8 about 2x, INT4 about 4x; AWQ and GPTQ; an AWQ INT4 model ran 52 to 194 tokens per second, 3.7x), and FlashAttention is a faster attention kernel. THE UNIFYING FRAME, three levers on the memory wall: batching amortizes the weight read across REQUESTS (throughput, L209), speculative decoding amortizes it across TOKENS in one sequence (latency, this lesson), and quantization makes the read SMALLER. Section roadmap strip: metrics, KV cache, batching, speculative decoding, serving with vLLM/TGI, streaming. Takeaway banner: speculative decoding trades cheap draft compute for fewer expensive forward passes, losslessly — pick a drafter whose guesses are accepted often, and remember it is a low-batch latency trick.

The Idea — Guess Ahead, Verify in One Pass

Normal decoding is strictly sequential: one forward pass → one token → feed it back → one pass → one token. Every token pays the full memory-bound cost of moving the whole model (L208). Speculative decoding breaks the one-pass-one-token rule with a draft → verify → accept loop:

Draft. A small, cheap drafter guesses the next K tokens given the text so far (e.g. it proposes the · cache · stores · past · keys). This is fast because the drafter is tiny (or it's just a lookup).
Verify. The big target model runs once, in parallel, over all K guessed positions — checking "is this what I would have said?" for each. Verifying K tokens in one pass is the key: it costs almost the same as generating a single token, because (L207–L208) the pass is memory-bound — it moves the model's weights out of memory regardless of whether it scores 1 token or 5.
Accept. Keep the matching prefix — every guess the target agrees with. At the first guess it disagrees with, throw away that token and the rest of the draft, and let the target emit its own correct token there. So this single pass commits (accepted + 1) tokens.

The best case: all K guesses are accepted, and the target also hands you a free (K+1)th token — K+1 tokens in one pass. The worst case: the very first guess is wrong, you commit 1 token (same as normal) but wasted the draft. Tokens-per-pass is the speedup, and it's driven by how often the drafter is right.

# Speculative decoding — one outer step = one TARGET forward pass
while not done:
    draft = drafter.generate(prefix, k=K)        # cheap: K guessed tokens
    target_logits = target(prefix + draft)        # ONE pass scores ALL K positions in parallel

    n_accept = 0
    for i, tok in enumerate(draft):
        if accept(tok, target_logits[i]):         # rejection sampling (lossless) — see below
            n_accept += 1
        else:
            break                                 # first disagreement: stop, discard the rest
    # commit the accepted prefix + ONE correct token sampled from the target at the break point
    prefix += draft[:n_accept] + [sample(target_logits[n_accept])]
    # committed (n_accept + 1) tokens for the price of ONE target pass

def tokens_per_pass(accept, K):
    """Expected tokens committed per TARGET forward pass (Leviathan et al. 2023).
    Geometric model: accept each of K draft tokens w.p. `accept`, plus 1 free
    token from the target.  = (1 - accept**(K+1)) / (1 - accept)."""
    return (1 - accept ** (K + 1)) / (1 - accept)

for accept in (0.4, 0.6, 0.8):
    row = [f"{tokens_per_pass(accept, K):.2f}x @K={K}" for K in (3, 5, 7)]
    print(f"accept={accept}:  " + "   ".join(row))
# Higher acceptance lets a bigger K pay off; at low acceptance, K barely helps.

It's Lossless — Rejection Sampling Preserves the Output

The part that surprises people: speculative decoding does not change the model's output at all. It is not an approximation, a smaller model, or a quality tradeoff. The text you get is drawn from exactly the same distribution as if the target model had decoded every token itself. The speedup is free of quality cost.

How? The acceptance rule is rejection sampling. For each drafted token, with the target's probability p and the drafter's probability q for that token:

Accept the draft token with probability min(1, p/q). If rejected, sample the replacement from the adjusted distribution norm(max(0, p − q)).

This is the classic trick for sampling from p using proposals from q. The math works out so that the final stream of tokens is distributed exactly as samples from the target p — provably identical. (For greedy decoding it's even simpler: accept the draft token iff it equals the target's argmax.)

Two consequences worth internalizing:

You can use any drafter — even a bad one — and still get correct output. A bad drafter just gets rejected more, so you get less speedup, never worse quality.
The drafter's only job is to be right often (high acceptance). It doesn't need to be good in general — just good at predicting this target here. That's why even a dumb n-gram lookup can work.

Where the Draft Comes From

Since any drafter is safe, the engineering question is just: what proposes good guesses cheaply? Four families, in rough order of "reach for it first":

Drafter	How it guesses	Accept	Speedup	Cost
n-gram / prompt lookup	copy the next tokens from a matching span in the prompt + output — zero model	30-80%*	1.3-3×	zero training / VRAM
Draft model	a small aligned model (e.g. 1B drafts for 70B) generates K tokens	60-82%	2.0-2.5×	a 2nd model in VRAM
Medusa	extra heads on the target predict several future tokens at once	50-65%	1.8-2.5×	light training
EAGLE / EAGLE-3	a small head predicts the target's hidden states + a tree of candidates	70-80%	3-4×	training; ~1-3 GB

*n-gram acceptance is workload-dependent: ~80%+ on repetitive / structured output, ~50-70% on code & JSON, but only ~20-40% on free-form chat.

The big practical lessons:

Try n-gram first. It's free and shockingly effective where the output reuses the input — RAG (quoting retrieved text), code (repeating identifiers), summarization & edits (copying spans). On those, prompt-lookup alone gives 2-3× for nothing.
EAGLE-3 is the 2025 state of the art for trained drafters (~3.5-4×), which is why production stacks ship it. Real H100 numbers on Llama-3.1-70B (batch 1): n-gram 1.5×, draft-1B 2.2×, EAGLE-2 3.0×, EAGLE-3 3.6×.
A drafter must share the target's tokenizer and be aligned to it — a mismatched drafter just gets rejected constantly (low acceptance, no speedup).

See It — The Spec Decode Lab

Step through the loop and feel where the speedup comes from. Pick a workload (which sets how often the draft is right) and the draft length K, then verify pass by pass — watch the green accepted prefix, the red rejected guess, the amber target correction, and the tokens-per-pass climb:

Step through speculative decoding. A cheap drafter proposes K tokens; the target verifies them all in ONE forward pass, accepts the matching prefix (green), rejects the first wrong guess (red), and emits one correct token of its own (amber) — committing 'accepted + 1' tokens per pass. Pick a workload (which sets how often the draft is right) and the draft length K, then watch tokens-per-pass — the speedup. Notice a longer K helps a LOT when acceptance is high (code/repetitive) but is wasted when it is low (creative chat), and the output is always identical to normal decoding. Illustrative; real speedups are about 2-4x.

Three things to take away:

Each pass commits (accepted + 1) tokens for the cost of one forward pass — that ratio is the speedup.
A longer K helps a lot when acceptance is high (code/repetitive → 3.5×+) but is wasted when it's low (creative chat stays ~1.8× no matter the K) — exactly why the rule of thumb is acceptance > 0.6, else lower K.
The output never changes across any setting — only the number of passes does. Speculation buys speed, not different text.

The Catch — Acceptance Rate, K, and Why Batching Fights It

Speculative decoding isn't free of tradeoffs — it's free of quality tradeoffs. Two real constraints decide whether it actually helps:

1. It lives and dies by the acceptance rate. Every drafted token costs a little draft compute whether or not it's accepted. If acceptance is high, the saved target passes dwarf that cost. If it's low, you're paying to draft tokens that get thrown away — and a bigger K makes it worse, not better. The practitioner's rule: aim for acceptance > 0.6 at K≈5; if you're below ~0.4, shrink K or drop speculation. "More draft" only pays if the draft is trusted.

2. It's a low-batch trick — and that puts it in tension with batching (L209). Speculative decoding cashes in the GPU's idle compute during memory-bound decode. But batching's whole job is to fill that idle compute with other requests. So once you're at high concurrency (batch ≳ 16), the GPU is already compute-bound — there's no spare compute for verifying speculative tokens, and the speedup collapses. The common guidance is blunt: at high batch, skip speculation.

This is the elegant duality of the section: batching fills idle compute with more requests (throughput, high concurrency); speculative decoding fills it with future tokens of the same request (latency, low concurrency). They draw from the same well of idle compute — so you use speculation when you have few users (or one, on-device) and batching when you have many. Two levers, opposite operating points.

Other Tricks — Shrink the Read & Speed the Kernels

Speculative decoding amortizes the weight read across tokens. The most important other trick attacks the memory wall from a different angle — it makes the read smaller.

Quantization. Store the model's weights in fewer bits — INT8 / FP8 (2× smaller than FP16) or INT4 (4× smaller). Since decode is memory-bound, fewer bytes to move per token ≈ proportionally faster (and cheaper, and it fits on smaller GPUs). The methods: AWQ (activation-aware, the current PTQ state of the art), GPTQ, SmoothQuant; FP8 is the usual first choice (best speed/accuracy). A real number: a Llama-2-7B running INT4 AWQ went from 52 → 194 tokens/s on one RTX 4090 — a 3.7× speedup — at a small, often-negligible quality cost. (You met these formats in fine-tuning, where the goal was fitting training in memory; here the goal is inference speed.) The cost: a little accuracy, and below 4 bits it degrades — so you measure (back to L207's evals) before shipping a quantized model.

Faster kernels. FlashAttention computes exact attention without ever materializing the big N×N attention matrix in slow memory — it fuses the operations and streams tiles through fast on-chip memory, cutting both memory use and time. It's on by default in modern stacks. (Kernel-level wins like CUDA graphs / torch.compile shave per-step launch overhead too.)

The three levers on the memory wall, together: batching spreads the weight read across requests (L209), speculative decoding spreads it across tokens (this lesson), and quantization makes the read smaller. They stack — a production server runs all three (e.g. AWQ weights + continuous batching + EAGLE at low load). The serving frameworks that wire them together are L211.

🧪 Try It Yourself

Reason through these, then confirm with the lab and the calculator:

Predict: in the lab, set Code/repetitive and raise K from 3 to 7. What happens to tokens-per-pass? Now do the same on Creative chat — why is it different?
Does speculative decoding change the text the model produces? Defend your answer in one sentence.
You're building a RAG assistant that quotes retrieved passages. Which drafter would you try first, and why is it especially good here?
Your speculative setup gives a great 3× speedup in dev (one user) but nothing in production under heavy load. What changed, and why?
Your drafter has only 35% acceptance. Name two fixes.

→ (1) On Code/repetitive (high acceptance) a longer K means many more tokens per pass (≈3.5×→4.7×); on Creative chat (low acceptance) K barely matters (~1.8×) because you hit a wrong guess early and discard the rest — bigger K just wastes draft work. (2) No — rejection sampling makes it lossless; the output is drawn from the exact target distribution. Speculation changes the number of forward passes, not the text. (3) n-gram / prompt-lookup — the answer reuses spans of the retrieved context, so copying the next tokens from the prompt hits a very high acceptance rate, for zero extra model or VRAM. (4) Production runs high batch, so the GPU is already compute-bound (batching filled the idle compute) — there's no spare compute for verifying speculative tokens, so the speedup vanishes. Speculation is a low-batch trick. (5) 35% is below the ~0.4 floor: (a) use a better-aligned drafter (EAGLE-3, or a draft model sharing the tokenizer / instruction style), and (b) lower K (fewer wasted drafts) — or switch to n-gram if the workload has prompt overlap, or just disable speculation.

Mental-Model Corrections

"Speculative decoding approximates the output / lowers quality." No — it's lossless. Rejection sampling makes the output identical to normal decoding from the target. A bad drafter costs speed, never quality.
"The draft model has to be good." It only has to be right often for this target (high acceptance). Even a trivial n-gram lookup works; a wrong guess is just rejected.
"It generates K tokens per pass." It generates (accepted + 1) per pass — somewhere from 1 (first guess wrong) to K+1 (all accepted). Tokens-per-pass = the speedup.
"Bigger K is always faster." Only if acceptance stays high. With low acceptance, a bigger K just wastes draft compute. Tune K to the acceptance rate.
"Turn it on everywhere for free throughput." It's a latency trick for low batch. At high concurrency the GPU is compute-bound (batching took the idle compute), and speculation stops helping — often skip it there.
"Quantization is just for fitting models on small GPUs." For inference it's also a speed lever — fewer weight bytes to move per token means faster memory-bound decode (and lower cost).
"These tricks compete." They stack. Quantized weights + continuous batching + speculative decoding (at low load) run together — three angles on the one memory wall.

Key Takeaways

Speculative decoding = guess ahead, verify in one pass. A cheap drafter proposes K tokens; the target verifies all K in one memory-bound pass and commits (accepted + 1) tokens. Tokens-per-pass is the speedup (~2-4×).
It's lossless. Rejection sampling (accept w.p. min(1, p/q)) makes the output identical to normal decoding. A weak drafter loses speed, never quality.
Drafters, easiest first: n-gram / prompt-lookup (zero cost, great for RAG/code/summarization where output reuses input) → draft model (small aligned model) → EAGLE-3 (2025 SOTA trained head, ~3.5-4×).
Speedup tracks acceptance. Aim > 0.6 at K≈5; below ~0.4, lower K or drop it. A bigger K only pays when the draft is trusted.
It's a low-batch / latency trick — and it fights batching. Both cash in the GPU's idle compute; at high concurrency you're compute-bound, so skip speculation and let batching work.
Other levers on the memory wall: quantization (INT8/FP8/INT4 via AWQ/GPTQ — smaller weight read → faster + cheaper) and FlashAttention (faster kernel). Batching (requests) · speculation (tokens) · quantization (smaller read) stack.
Next — L211: Serving with vLLM / TGI — the frameworks that bundle the KV cache, PagedAttention, continuous batching, speculative decoding, and quantization into one server you actually run.