Parallelization

Introduction

Pattern 3 of 5 (L115). Chaining (L116) ran steps in sequence; routing (L117) picked one path. Parallelization does the opposite of "one at a time": it runs many LLM calls at once and combines their results. In Anthropic's words: "LLMs can sometimes work simultaneously on a task and have their outputs aggregated programmatically."

There are two distinct reasons to do this, and Anthropic names both:

Sectioning — split a task into independent subtasks and run them in parallel, for speed.
Voting — run the same task several times and combine the answers, for confidence.

They look similar (many calls at once) but optimize for different things — and they fail in different ways. By the end you'll know exactly when to reach for each, the latency-vs-cost math that makes parallelization worth it, and the single most dangerous trap in the whole pattern: voting that's confidently wrong.

In this lesson:

Sectioning and voting from scratch (it's asyncio.gather — a few lines)
The latency math: parallel = the slowest call, not the sum — and why cost doesn't drop
Aggregation strategies (the genuinely hard part)
The voting trap — consensus is not verification — and the production realities (rate limits, failures, the Batch API)
When not to parallelize, and the framework (LangGraph's Send)

An infographic titled 'Parallelization — run many calls at once, then aggregate'. LLMs can work simultaneously on a task and have their outputs aggregated programmatically. TWO VARIATIONS. Sectioning (for speed): break a task into INDEPENDENT subtasks and run them at once — review a pull request along four axes (Security, Performance, Tests, Style), each its own focused LLM call, then merge into one report; wall-clock latency is the SLOWEST call, not the sum, so you get four reviews in the time of one. Voting (for confidence): run the SAME task several times (three differently-phrased safety prompts) and combine the votes with an any-unsafe or majority threshold. THE LATENCY AND COST MATH: sequential = the sum of every call; parallel = about the max (the slowest straggler); N times t becomes about 1 times t. But token COST is unchanged for sectioning and N times higher for voting — you trade money and throughput for lower latency or higher confidence, never the reverse. AGGREGATION is the hard part: concatenate/merge, majority/consensus, an any-OR gate (flag if any branch flags — high recall), or an LLM synthesizer to reconcile conflicts. THE HONEST FAILURE: voting amplifies CORRELATED ERRORS — lookalike voters share blind spots, so the majority can be confidently wrong (on MATH-style tasks 65 to 87 percent of incorrect answers collapse onto a single wrong answer); fix by decorrelating voters and adding an external verifier. Fan-out also hits rate limits and partial failures, so use bounded concurrency (a semaphore), retries with backoff, and return_exceptions. USE IT WHEN subtasks are independent (parallelize for speed) or you need multiple attempts/perspectives (vote for confidence); NOT when outputs depend on each other (that is prompt chaining), the task is atomic, or the voting tax is unjustified. In Python run concurrent calls with asyncio.gather or a ThreadPoolExecutor (LLM calls are I/O-bound, so the GIL is not the bottleneck). Takeaway: fan independent work out for speed or run it N times for confidence, then aggregate; you pay in tokens not wall-clock, and voting only adds truth when voters are decorrelated and a verifier checks them.

Two Flavors: Sectioning vs Voting

The two variations share a shape but answer different questions. Memorize both definitions — they're Anthropic's, verbatim:

✂️ Sectioning — "Breaking a task into independent subtasks run in parallel." The subtasks are different and independent; you run them concurrently and stitch the pieces together. Anthropic's examples: guardrails ("one model instance processes user queries while another screens them for inappropriate content" — which "tends to perform better than having the same LLM call handle both guardrails and the core response"), and automated evals ("each LLM call evaluates a different aspect").

🗳️ Voting — "Running the same task multiple times to get diverse outputs." The calls are the same task; you combine their answers into a more confident one. Anthropic's examples: code review for vulnerabilities ("several different prompts review and flag the code if they find a problem") and content moderation ("multiple prompts evaluating different aspects or requiring different vote thresholds to balance false positives and negatives").

And the headline when-to-use, verbatim:

"Parallelization is effective when the divided subtasks can be parallelized for speed, or when multiple perspectives or attempts are needed for higher confidence results. For complex tasks with multiple considerations, LLMs generally perform better when each consideration is handled by a separate LLM call, allowing focused attention on each specific aspect."

That last sentence is the deep reason sectioning improves quality, not just speed: a reviewer thinking about only security is sharper than one juggling security, performance, style, and tests at once.

Sectioning From Scratch

Sectioning is just concurrent independent calls + a merge. In Python, asyncio.gather fires them all at once and waits for the lot, preserving input order:

import asyncio
from anthropic import AsyncAnthropic
client = AsyncAnthropic()

# SECTIONING: four INDEPENDENT reviewers, one narrow prompt each.
REVIEWERS = {
    "security":    "Review ONLY for security vulnerabilities. Be specific.",
    "performance": "Review ONLY for performance problems. Be specific.",
    "tests":       "Review ONLY for missing test coverage. Be specific.",
    "style":       "Review ONLY for style & readability nits. Be specific.",
}

async def review_one(system: str, diff: str) -> str:
    msg = await client.messages.create(
        model="claude-sonnet-4-5", max_tokens=512,
        system=system, messages=[{"role": "user", "content": diff}],
    )
    return msg.content[0].text

async def review_pr(diff: str) -> dict[str, str]:
    # Fire all four AT ONCE. gather() awaits them and preserves input order.
    results = await asyncio.gather(*(review_one(s, diff) for s in REVIEWERS.values()))
    return dict(zip(REVIEWERS, results))          # <- programmatic aggregation
# Wall-clock time = the SLOWEST reviewer, not the sum of all four.

Read it: four review_one(...) calls, each a narrow, focused prompt, all launched together by gather. The return is a plain dict — that's your programmatic aggregation (here, just labeling each result). The key property: because the four are independent, the wall-clock time is the slowest single reviewer, not the sum of all four.

Don't have an async codebase? Threads are just as good here, because LLM calls are I/O-bound — you're waiting on the network, so Python's GIL is released and many calls run "in flight" at once. That's exactly what Anthropic's cookbook does:

# No async in your codebase? THREADS work just as well — LLM calls are I/O-bound
# (you're waiting on the network), so the GIL is released while each call blocks.
# This is exactly what Anthropic's cookbook `parallel()` does:
from concurrent.futures import ThreadPoolExecutor

def parallel(prompt: str, inputs: list[str], n_workers: int = 3) -> list[str]:
    with ThreadPoolExecutor(max_workers=n_workers) as executor:
        futures = [executor.submit(llm_call, f"{prompt}\nInput: {x}") for x in inputs]
        return [f.result() for f in futures]      # .result() preserves input order
# Threads OR asyncio — both let many requests be "in flight" at once. (For CPU-bound
# work you'd need multiprocessing; for LLM fan-out you never do.)

The Latency Math (the whole point)

Here is the entire economic argument for sectioning, in two lines:

Sequential latency = Σ — the sum of every call's latency.
Parallel latency ≈ max — the slowest single call (the "straggler"), plus a little overhead.

So N independent calls of ~t each go from N×t down to ~1×t. Four ~2-second reviews: ~8s sequential → ~2s parallel. That's the win, and it's huge for anything user-facing.

But here's the part beginners get wrong — be honest about what you're trading:

Token cost is unchanged for sectioning. You still make the same four calls; you just made them concurrently. Same tokens, same dollars — only the clock changed.
Token cost is N× higher for voting. You deliberately re-run the same question, so 5 votes ≈ 5× the cost of one.
You never save money by parallelizing. You trade money and throughput for lower latency (sectioning) or higher confidence (voting). Parallelization optimizes latency and quality — never cost.

One more subtlety: because parallel time is the slowest call, tail latency matters. A single slow call drags the whole batch — so add per-call timeouts (asyncio.wait_for), and watch for stragglers.

Voting From Scratch

Voting is the same call N times, then a threshold. The two classic aggregators are any (flag if any voter flags — maximizes recall) and majority (balances false positives and negatives):

# VOTING: run the SAME check N times with DIFFERENT phrasings, then threshold.
SAFETY_PROMPTS = [
    "Does this violate our content policy? Answer SAFE or UNSAFE.",
    "Is this harmful, deceptive, or illegal? Answer SAFE or UNSAFE.",
    "Would you be comfortable showing this to a child? Answer SAFE or UNSAFE.",
]

async def is_safe(text: str) -> bool:
    votes = await asyncio.gather(*(review_one(p, text) for p in SAFETY_PROMPTS))
    unsafe = sum("UNSAFE" in v.upper() for v in votes)
    return unsafe == 0                # ANY-unsafe gate -> high recall (flag if ANY flags)
    # majority would be:  return unsafe <= len(votes) // 2
# The DIFFERENT phrasings are the point: they DECORRELATE the voters. Three identical
# prompts just reproduce one answer three times — and agree on the same mistake.

Voting has deep roots: self-consistency (Wang et al., 2022) samples multiple reasoning chains at temperature > 0 and takes the majority answer — the idea being that a correct answer is reachable by many paths, so it accumulates votes while wrong answers scatter. It lifted GSM8K by +17.9%, SVAMP +11.0%, AQuA +12.2% over plain chain-of-thought. (Caveat: those gains were on 2022 models; on a modern reasoning model whose single-pass CoT is already strong, the lift is much smaller.)

Two practical rules:

Diminishing returns. Cost grows linearly with N; accuracy gains are sublinear and saturate — most of the benefit is in the first few votes (≈3–10). Pushing N to 40 burns money for crumbs.
Decorrelate the voters. Different phrasings (or different models, or different framings) are what make voting work. Three identical prompts don't give you three opinions — they give you the same opinion three times. Which leads to the trap…

See It Run (Sectioning & the Voting Trap)

Switch between Anthropic's two variations below. In Sectioning, toggle Sequential vs Parallel and watch the timeline bars glide from a staircase (the sum) to a stack (the slowest call). In Voting, run the safety check — then switch to the "correlated voters" scenario and watch what unanimous agreement is really worth:

Two tabs for Anthropic's two variations. SECTIONING: a PR review fans out to 4 independent dimension-workers — toggle Sequential vs Parallel and watch the Gantt bars GLIDE from a staircase (sum of latencies) to a left-aligned stack (the slowest call), while token cost stays identical. VOTING: run the same safety check through 3 voters with an any/majority threshold — then switch to the 'correlated voters' scenario and watch 3/3 unanimous 'safe' votes be confidently WRONG, because lookalike voters share the same blind spot (agreement tracks shared priors, not truth).

The sectioning tab makes the latency win physical: same four calls, same cost, but the Parallel clock lands on the slowest call instead of the sum. The voting tab sets up the next section — flip to correlated voters and you get 3/3 "safe", high confidence… and confidently wrong. That's not a bug in the widget; it's the deepest gotcha in the whole pattern.

Aggregation: The Hard Part

Spawning the calls is easy; combining their outputs is where the real engineering is. Pick the aggregator deliberately — it encodes your priorities:

Concatenate / merge — stitch independent sections into one deliverable (the PR report). The natural choice for sectioning.
Majority / consensus — take the most common answer. For voting on discrete answers (a label, a number).
Any / OR-gate — if any call flags it, flag it. Maximizes recall — the right call for safety/vulnerability screening, where a miss is costlier than a false alarm. (An AND-gate / unanimity does the reverse: maximizes precision.)
LLM synthesizer — an extra LLM call reads all N outputs and writes one coherent answer. Use it when outputs conflict or overlap and need reconciling (this is the core of Mixture-of-Agents, which beat GPT-4o on AlpacaEval 2.0 using only open models). It costs an extra call and is itself a failure point.
Best-of-N / take-best — score the candidates (heuristic, reward model, or LLM judge) and keep the single best.

Watch the seams. Sectioning can leave gaps (something falls between two sections) or duplicates (two sections both claim it). Voting produces real disagreement the aggregator must resolve, not paper over. Bad merge logic is the quiet #1 source of parallelization bugs.

The Voting Trap: Consensus Is Not Verification

This is the most important — and most counterintuitive — idea in the lesson. Voting amplifies agreement, which is not the same as truth.

Why: LLMs share training data, objectives, and post-training incentives, so they share blind spots — their errors are correlated. When the dominant answer is wrong, extra samples and extra voters mostly reproduce the same mistake. A 2026 Stanford study ("Consensus Is Not Verification") quantified it: voting "yields no consistent accuracy gains over single-sample baselines and often amplifies shared misconceptions" in domains lacking an external checker. On MATH problems, 65–87% of incorrect answers collapse onto a single wrong answer — so the "majority" confidently agrees on the same error. And cranking temperature barely helps: only ~2.9% of plurality answers changed between T=0.7 and T=1.0. Confidence is no rescue either — it "tracks expected consensus rather than correctness."

So when does voting add truth?

When the voters are genuinely decorrelated — different prompts, framings, or models — so they don't share the same blind spot. (Temperature alone is too weak.)
When there's an external verifier — a math answer-checker, code that actually runs and passes tests, a retrieval check. Voting + a verifier is powerful; voting alone on open-ended factual questions is a confidence machine, not a correctness machine.

The one-liner to carry forever: agreement among lookalike voters is shared bias, not evidence. That's exactly what the widget's "correlated voters" scenario shows.

Production Reality + The Honest Tradeoff (and When NOT to Parallelize)

Naive fan-out (gather over 10,000 prompts at once) immediately trips provider rate limits (RPM/TPM → 429s), and any one call can fail or hang. Production parallelization needs guardrails:

# Real fan-out trips rate limits (429s) and the occasional call fails or hangs.
# Bound the concurrency, and don't let one bad call sink the batch.
sem = asyncio.Semaphore(10)                        # at most 10 calls in flight at once

async def bounded(system: str, diff: str) -> str:
    async with sem:                                # the semaphore caps concurrency;
        return await review_one(system, diff)      # the SDK already retries 429/5xx w/ backoff

results = await asyncio.gather(
    *(bounded(s, diff) for s in REVIEWERS.values()),
    return_exceptions=True,                         # a failed call comes back as an Exception,
)                                                   # NOT a crash of the whole gather
ok = [r for r in results if not isinstance(r, Exception)]   # 3 of 4 still makes a report
# Not latency-sensitive (bulk evals, moderation sweeps)? Use the Batch API instead:
# ~50% cheaper, results within 24h, on a separate rate-limit pool.

The costs, stated plainly:

Cost multiplication. N parallel calls = ~N× tokens — brutal for voting, which re-runs the same question.
Rate limits & partial failures. Bound concurrency with a semaphore, retry 429/5xx with exponential backoff + jitter (the SDKs do this for you), set per-call timeouts, and pass return_exceptions=True so one bad call doesn't sink the batch. Not latency-sensitive? The Batch API (Anthropic Message Batches / OpenAI Batch) is ~50% cheaper, within 24h, on a separate rate-limit pool — the right tool for bulk evals and moderation sweeps.
Aggregation bugs — the seams and ties from the last two sections.

When NOT to parallelize:

The subtasks aren't independent. If B needs A's output, that's chaining (L116) — running them "in parallel" either loses the dependency or fabricates it. Sectioning requires genuinely independent parts.
The task is a single atomic step, or latency is already fine.
Low stakes + voting cost. If an occasional error is cheap (e.g. alt-text generation), the N× tokens aren't justified.

Keep the family straight: Parallelization runs many known subtasks at once and aggregates. Routing (L117) picks one of N. Orchestrator-Workers (L119, next) also fans out — but its "subtasks aren't pre-defined; [they're] determined by the orchestrator based on the specific input." Parallelization's subtasks are fixed and known; the orchestrator's are dynamic.

Then — and Only Then — the Framework

You built fan-out with asyncio.gather, so a framework will never be a black box. LangGraph makes parallelization a first-class graph operation — multiple edges from one node run concurrently, and the Send API spawns a dynamic number of parallel branches (map-reduce), with a reducer merging the results on fan-in:

# THEN a framework. LangGraph fans out with the Send API (dynamic map-reduce):
from langgraph.types import Send
from typing import Annotated
import operator

def fan_out(state):            # one parallel branch per item (count unknown at compile time)
    return [Send("review", {"aspect": a, "diff": state["diff"]}) for a in state["aspects"]]

builder.add_conditional_edges("plan", fan_out, ["review"])

class State(TypedDict):        # fan-IN: a reducer MERGES the parallel branches' outputs
    findings: Annotated[list, operator.add]        # branches APPEND, they don't overwrite
# (LCEL's RunnableParallel / .abatch(max_concurrency=...) do the same for static fan-out.)

Send is the map (one branch per item); the Annotated[list, operator.add] reducer is the reduce (branches append to shared state instead of clobbering it). In plain LCEL, RunnableParallel runs a dict of branches concurrently and .abatch(max_concurrency=...) fans out over a list of inputs.

One sibling idea worth naming: parallel tool calls — the model emitting several tool_use blocks in a single turn that you execute concurrently (OpenAI's parallel_tool_calls, Claude's multiple tool_use blocks). That's concurrency within one model turn, distinct from the workflow pattern (you orchestrating multiple model calls) — but it rhymes. Build the fan-out once by hand; the framework just adds the plumbing. (Same lesson as L111 — Building a ReAct Agent From Scratch/L116 — Prompt Chaining/L117 — Routing.)

🧪 Try It Yourself

Reason through these, then use the widget to confirm:

Predict before toggling: in the widget's Sectioning tab, four reviewers take 2.2s, 1.6s, 1.9s, 1.1s. What's the wall-clock time sequential vs parallel, and what's the token cost of each?
Your pipeline does: transcribe audio → summarize the transcript → translate the summary. A teammate wants to "parallelize it for speed." What's wrong with that, and what could you parallelize here?
You run a safety classifier 5× and majority-vote. It reports 5/5 agree, 98% confident that a post is safe — but it's actually a scam. What most likely happened, and what are the two fixes?
You're moderating user posts where missing a harmful one is far worse than a false alarm. Which aggregation rule do you pick — any or majority — and why?
When is voting a waste of money — i.e., when does running the same task N times not buy you anything?

→ (1) Sequential = 2.2+1.6+1.9+1.1 = 6.8s; parallel ≈ 2.2s (the slowest call). Token cost is identical — four calls either way; you bought time, not money. (2) Those steps are a chain, not parallel work — each depends on the previous one's output (you can't translate a summary you haven't written). It's prompt chaining, and parallelizing dependent steps is a category error. What you could parallelize: if you had many audio files, run the whole chain over each file concurrently (map-reduce). (3) Correlated errors — five identical (or near-identical) voters share the same blind spot, so they agree on the same wrong answer; the 98% "confidence" tracks their shared bias, not truth. Fixes: (a) decorrelate — diverse prompts/framings/models; (b) add an external verifier (a rule, a checker, retrieval) — voting alone isn't verification. (4) Any (flag if any voter flags) — it maximizes recall, which is what you want when a miss is the costly error; majority could out-vote the one voter that caught it. (5) When the voters are correlated (they'll just repeat one answer), when there's no verifier and the task is open-ended (consensus ≠ truth), when the task is already reliable single-pass, or when the stakes don't justify the N× cost.

Mental-Model Corrections

"Parallelization saves money." Never. It saves wall-clock latency (sectioning) or buys confidence (voting) — you still pay for every call, and voting pays N×. You trade dollars for time, not the reverse.
"Parallel = the sum of the calls, just faster." No — parallel latency is the slowest single call (max), not the sum. That's the whole point.
"I can parallelize any multi-step task." Only independent steps. If B needs A's output, it's a chain — parallelizing it loses the dependency.
"More votes = more accuracy." Sublinear and saturating, and worthless if the voters are correlated — then extra votes just re-buy the same (possibly wrong) answer. Most of the gain is in the first few.
"If all the voters agree, it must be right." Consensus is not verification. Lookalike models share blind spots; on MATH, 65–87% of errors collapse to one wrong answer. Agreement = shared bias unless voters are decorrelated and checked by a verifier.
"The hard part is firing the calls." The hard part is aggregation — merges with gaps/dupes, voting ties, reconciling conflicts. Spawning is gather; combining is the engineering.
"Just gather over all 10k prompts." You'll get 429s. Bound concurrency with a semaphore, retry with backoff, return_exceptions=True, and use the Batch API when you don't need low latency.

Key Takeaways

Parallelization = run many LLM calls simultaneously and aggregate programmatically. Pattern 3 of 5. Two flavors: Sectioning (independent subtasks, for speed) and Voting (same task ×N, for confidence).
The latency math: sequential = Σ (sum); parallel ≈ max (the slowest call). N×t → ~1×t. Cost is unchanged for sectioning and N× for voting — you trade money/throughput for latency or confidence, never cost.
Build it from scratch: asyncio.gather (or a ThreadPoolExecutor — LLM calls are I/O-bound, so the GIL isn't the bottleneck). Each focused call is also sharper than one juggling everything.
Aggregation is the real work: concatenate (sections), majority (votes), any/OR (high recall), or an LLM synthesizer (reconcile). Mind the seams (gaps/dupes) and ties.
The voting trap — consensus is NOT verification: correlated voters share blind spots and agree on the same wrong answer (65–87% collapse on MATH). Voting adds truth only when voters are decorrelated and an external verifier checks them.
Production: bound concurrency (semaphore), retry+backoff, return_exceptions=True, timeouts; use the Batch API (~50% cheaper, 24h) when latency doesn't matter.
Don't parallelize dependent steps (that's chaining) or atomic/low-stakes tasks. Parallelization's subtasks are known; an orchestrator's are dynamic — which is exactly L119 (Orchestrator-Worker), next.