Parallel & Sequential Tool Calls

Introduction

You can design a perfect tool (L122) and still build a slow agent — if it calls those tools one at a time when it didn't have to. This lesson is about orchestration: when the model fires several tool calls in a single turn, how you run them decides your latency, and which calls must wait on others.

The whole lesson turns on one distinction:

Parallel — calls that are independent of each other can run at the same time. Latency collapses from the sum of the calls to the slowest one (the critical path).
Sequential — when call B needs the output of call A (a data dependency), or when a call has side effects that must be ordered, you have to run them in order.

You've met the idea before — the parallelization workflow pattern in L118 (Parallelization) fanned out independent subtasks. This is that idea at the tool-call level: the concrete API mechanics of multiple tool_use blocks, concurrent execution, and the one formatting rule that keeps it all working.

In this lesson:

One turn, many calls — how the model signals parallelism (multiple tool_use blocks)
Execution is your choice — concurrency, and why latency becomes the critical path
The golden formatting rule — and the #1 mistake that silently kills parallelism
Sequential = data dependency, when not to parallelize, and how to encourage it

Scope: this builds on the L121 (Tools) lifecycle and L122 (Designing Robust Tool Interfaces) interface craft. The code-execution sandbox is L124 (Code Execution & Sandboxing); error handling & retries are L125 (Handling Tool Errors & Retries) (we'll only touch the is_error shape here).

An infographic titled 'Parallel & Sequential Tool Calls'. The model can emit MANY tool_use blocks in a single assistant turn — and how you run them decides your latency. SEQUENTIAL: one call at a time, so wall-clock latency is the SUM of every call (0.8 + 1.4 + 1.1 + 0.5 seconds). PARALLEL: independent calls are batched and run concurrently (asyncio.gather, Promise.all), so latency collapses to the CRITICAL PATH — the longest chain — not the sum; benchmarks show roughly 1.8 to 3.7 times wall-clock speedups (the LLMCompiler showed up to 3.7x faster and 6x cheaper than sequential ReAct). But you cannot parallelize across a DATA DEPENDENCY: if convert_price needs find_hotel's result, it must wait — the dependency, not the slowest single call, sets the floor. THE EXECUTION CHOICE IS YOURS: the API does not prescribe order; run independent read-only calls in parallel for lower latency, and run tools with side effects, shared state, or ordering requirements sequentially. THE GOLDEN RULE OF FORMATTING: return exactly one tool_result for each tool_use block, and put them ALL in a SINGLE user message (a call you skipped or that failed still gets a tool_result with is_error true). Splitting the results across separate user messages silently TEACHES the model to stop calling tools in parallel — the most common reason parallelism mysteriously disappears. CONTROL: disable_parallel_tool_use true forces at most one tool per turn; models can under-use parallelism, so a system-prompt nudge ('whenever you perform multiple independent operations, invoke all relevant tools simultaneously') and measuring the average number of tools per tool-calling message (it should exceed 1.0) push it up. Takeaway: many tool_use blocks per turn; run the independent ones concurrently so latency is the critical path not the sum; serialize true data dependencies and side-effecting writes; and always return every tool_result in one user message.

One Turn, Many Calls

In L121 (Tools) every example called one tool per turn. But nothing limits the model to one — a single assistant message can contain several tool_use blocks at once. That is parallel tool use, and on Claude 4 models it happens by default whenever the requested operations are independent.

import anthropic
client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-4-8", max_tokens=1024, tools=tools,
    messages=[{"role": "user",
               "content": "What's the weather in SF and NYC, and the time in each?"}])

# A SINGLE assistant message can carry MANY tool_use blocks — that's parallelism.
tool_uses = [b for b in response.content if b.type == "tool_use"]
print(len(tool_uses))          # -> 4   (NOT one tool per turn)
# Claude 4 models do this by default when the calls are independent.

Ask for the weather in SF and NYC and the time in each, and the model doesn't make four sequential round-trips — it emits four tool_use blocks in one response. Your signal that parallelism is happening is simply len(tool_uses) > 1. The model has done the planning ("these four are independent"); what it hands you is a batch.

Execution Is Your Choice — and Latency Is the Critical Path

Here's the part the API is deliberately quiet about:

"When Claude returns multiple tool_use blocks in a single assistant turn, how you run them is your decision. The API doesn't prescribe an execution order."

You can run them concurrently (asyncio.gather, Promise.all), sequentially, or any mix. For independent, read-only calls, concurrency is the win:

import asyncio

# The API does NOT prescribe order. Independent, read-only calls -> run concurrently.
async def run_batch(tool_uses):
    async def one(b):
        out = await execute(b.name, b.input)          # your real tool
        return {"type": "tool_result", "tool_use_id": b.id, "content": str(out)}
    return await asyncio.gather(*[one(b) for b in tool_uses])   # all at once

tool_results = asyncio.run(run_batch(tool_uses))

messages.append({"role": "assistant", "content": response.content})
messages.append({"role": "user", "content": tool_results})   # ALL results, ONE message
# Wall-clock latency is now max(call times), not their sum.
# LLMCompiler measured up to 3.7x faster / 6x cheaper than a sequential ReAct loop.

Why it matters: "when tools involve I/O — API calls, database queries, entire agent conversations — concurrent execution means the total time is determined by the slowest operation, not the sum." Four 1-second calls run sequentially take ~4s; run together they take ~1s.

This is the single biggest latency lever in agent engineering. Production benchmarks show ~1.8×–3.7× wall-clock speedups from scheduling independent work concurrently, and the LLMCompiler (Kim et al., 2023 — a Planner + parallel Executor) measured up to 3.7× faster and 6× cheaper than a sequential ReAct loop. Same model, same tools — just better scheduling.

The Golden Rule: All Results in One Message

There is one formatting rule you must not break, and it's the most common cause of "parallelism mysteriously stopped working":

Return exactly one tool_result per tool_use block — and put them ALL in a single user message.

# ✅ KEEPS parallel tool use — every result in ONE user message
[{"role": "assistant", "content": [tool_use_1, tool_use_2]},
 {"role": "user",      "content": [tool_result_1, tool_result_2]}]

# ❌ TEACHES the model to STOP parallelizing — results split across messages
[{"role": "assistant", "content": [tool_use_1, tool_use_2]},
 {"role": "user",      "content": [tool_result_1]},      # one...
 {"role": "user",      "content": [tool_result_2]}]      # ...per message

# A call you skipped or that failed STILL needs a tool_result:
{"type": "tool_result", "tool_use_id": "toolu_02", "is_error": True,
 "content": "Not executed: the preceding write_file call failed."}

Why does splitting them matter? Because the conversation history is also a training signal for the next turn. When the model sees its past parallel batch answered in separate user messages, it learns to stop batching — Anthropic is explicit that this "teaches Claude to avoid parallel calls." One assistant message of tool_use blocks → one user message of tool_result blocks.

Two corollaries:

Every tool_use needs a matching tool_result, even ones you skipped or that failed — return is_error: true with a brief reason. A missing result is an API error.
No text before the tool results in that user message — just the tool_result blocks.

The widget's RESULT FORMATTING toggle lets you flip between the ✓ and ✗ shapes and read exactly what each one does to future parallelism.

Sequential = a Data Dependency

Parallelism has a hard limit: you cannot run a call before the call it depends on. If convert_price needs the price returned by find_hotel, then find_hotel must finish first. That chain — find_hotel → convert_price — is a data dependency, and it forces sequential execution for those two no matter how much you'd like to parallelize.

This is why parallel latency is the critical path, not a flat "slowest single call." In the widget's trip example, three calls are independent (run together) but the dependent chain (find_hotel 1.1s → convert_price 0.5s = 1.6s) is what actually bounds the turn — longer than the slowest independent call (search_flights, 1.4s). The longest dependency chain sets the floor.

How does the model decide? It plans the dependency structure itself — it will chain calls across turns when a later call needs an earlier result (the ReAct loop from L111 — Building a ReAct Agent From Scratch: act → observe → act again), and batch them in one turn when they're independent. You can help it keep the two straight:

"To reduce dependent calls appearing together, add to your system prompt: 'Only batch tool calls that are independent of each other.'"

If you do run a batch sequentially and an earlier call fails, stop and return is_error: true for the ones you didn't run — the model will reissue them next turn.

See It — The Tool Scheduler

Below is the trip-briefing example as a live scheduler. Toggle Sequential ↔ Parallel and watch the Gantt, the wall-clock latency, and the speedup; then toggle the result-formatting rule:

“Brief me for my Tokyo trip” fans out into four tool calls — three independent (weather, flights, hotel) and one dependent (convert_price needs the hotel's price). Flip EXECUTION between Sequential and Parallel and watch the Gantt animate: sequential stacks the bars end-to-end so latency is the SUM (3.8s); parallel starts the independent calls together, but the dependent call still has to wait for find_hotel — so latency collapses to the CRITICAL PATH (find_hotel → convert_price = 1.6s), not a single max, for ~2.4× faster. Then flip RESULT FORMATTING to feel the #1 gotcha: returning all tool_results in ONE user message keeps the batch alive, while splitting them across separate messages silently trains the model to stop parallelizing.

Three things to take away from playing with it:

Parallel doesn't mean instant — it means latency = the critical path. The dependent convert_price still waits for find_hotel, so the floor is 1.6s, not the 1.4s of the slowest independent call.
The independent three (weather, flights, hotel) all start at t=0 — that's where the speedup comes from.
Splitting results flips the message panel red and explains the silent cost: you've just taught the model to stop batching.

When NOT to Parallelize

Concurrency is the default instinct for reads, but it's the wrong instinct for writes. Run calls sequentially when:

There's a data dependency — B consumes A's output. (The structural case above.)
Calls have side effects / shared state. Two tools that both write the same record, increment a counter, or append to a file can race and corrupt state. "Tools with side effects, shared state, or ordering requirements might be better run sequentially."
Order is semantically required — create_order then charge_card then send_confirmation must happen in that order; parallelizing them is nonsense.
The actions aren't idempotent — a parallel retry of a non-idempotent write can double-charge or double-send.
You'd blow a rate limit or a connection pool — firing 50 calls at once can get you throttled; bound concurrency (a semaphore) even for independent reads.

The clean heuristic: independent reads → parallelize; ordered or side-effecting writes → serialize. When unsure, serialize — a slow correct agent beats a fast corrupt one.

Encouraging Parallelism (Models Under-Use It)

Even on capable models, the default rate of parallel calls is often lower than optimal — the model plays it safe and sequences things it could have batched. Three levers:

1. Prompt for it. A short system-prompt nudge measurably raises the parallel rate; Anthropic ships a recommended block:

# Models can UNDER-use parallelism. Nudge it in the system prompt:
SYSTEM = """<use_parallel_tool_calls>
Whenever you perform multiple INDEPENDENT operations, invoke all relevant tools
simultaneously rather than sequentially. e.g. to read 3 files, make 3 tool calls
in one turn. Only batch tool calls that are independent of each other.
</use_parallel_tool_calls>"""

# Force at-most-one tool per turn when you must (e.g. risky writes):
client.messages.create(..., tool_choice={"type": "auto",
                                          "disable_parallel_tool_use": True})

# Measure it: average tool_use blocks per tool-calling message should be > 1.0.

2. Phrase user requests to invite it. "Check the weather in Paris and London simultaneously" batches better than two separate sentences.

3. Measure and disable when needed. Track the average number of tool_use blocks per tool-calling message — it should be > 1.0 if parallelism is working (it's the quickest health check). And when you must force one-at-a-time — risky writes, strict ordering — set disable_parallel_tool_use: true (with tool_choice: auto it caps at at most one tool; with any/tool it forces exactly one).

The fastest way to lose parallelism is the formatting bug from earlier; the fastest way to gain it is the system-prompt block above. Check both before concluding "the model just won't parallelize."

The Frontier — DAG Scheduling & Programmatic Calls

Native parallel tool use batches the calls the model can see are independent in one turn. The research frontier pushes further, and it's worth knowing the shape of it:

Plan-as-a-DAG (LLMCompiler). Instead of letting the agent discover dependencies turn by turn, a Planner emits the whole task graph up front, a Task-Fetching Unit resolves dependencies, and an Executor runs every independent node concurrently. This is the orchestrator-worker idea (L119 — Orchestrator-Worker) applied to tool calls, and it's where the 3.7× / 6× numbers come from.
Programmatic tool calling. For many chained calls with large intermediate results, the model can write code that calls the tools in a loop/branch (running in the code-execution sandbox), so the orchestration happens in real control flow and only the final result returns to the context. (That sandbox is L124 — Code Execution & Sandboxing.)
Speculative / async execution. Newer techniques (e.g. PASTE, 2026) start likely-needed calls before the model formally requests them, hiding even more latency.

You don't need these to ship — native parallel + a clean dependency split covers the vast majority of cases. But knowing the ceiling tells you where to reach when latency is your bottleneck and the dependency graph is rich.

🧪 Try It Yourself

Reason through these, then use the scheduler to confirm:

Predict: four independent read calls of 1s each. Sequential latency? Parallel latency? Now make the 4th depend on the 3rd — what's the new parallel latency?
Your agent calls tools in parallel for a while, then "randomly" goes one-at-a-time. You changed nothing about the prompt. What's the most likely cause?
The model wants to run update_inventory(item=A, -1) and update_inventory(item=A, -1) in parallel. Should you let it? Why / why not?
You have get_user, then get_orders(user.id), then summarize(orders). How much of this can be parallelized?
The model keeps sequencing calls that are clearly independent. Name two fixes.

→ (1) Sequential = 4s (the sum); parallel = ~1s (they all run at once, latency = slowest = 1s). Make #4 depend on #3 and parallel becomes ~2s — the dependent chain (#3→#4 = 2s) is the critical path, longer than any single call. (2) You're splitting tool_results across separate user messages — that teaches the model to stop batching. Put all results in one user message. (3) No — same record, a side-effecting non-idempotent write; running them in parallel can race and lose an update. Serialize side effects. (4) None of it — it's a pure dependency chain (get_user → get_orders → summarize); each needs the previous result, so it's inherently sequential. (5) Add the <use_parallel_tool_calls> system-prompt block, and/or phrase the request to ask for simultaneous calls; also verify you're returning results in one message (the formatting bug suppresses parallelism).

Mental-Model Corrections

"One tool call per turn." A single assistant message can carry many tool_use blocks — that's parallel tool use, on by default for Claude 4.
"The API runs my tools." It returns the requests; you execute them — and you choose concurrent vs sequential. Concurrency is an asyncio.gather, not a model setting.
"Parallel latency = the slowest call." Only if everything's independent. With a dependency, it's the critical path — the longest chain.
"I can return tool results however I like." No — all results go in one user message, one per tool_use (including is_error for skipped/failed). Splitting kills future parallelism.
"More parallelism is always better." Not for writes — side effects, ordering, and non-idempotent actions need serial execution; unbounded fan-out can trip rate limits.
"If the model isn't parallelizing, it can't." Usually it's weak prompting or the formatting bug. Add the system-prompt block; check the message shape; measure tools-per-message.
"Sequential chaining is a failure." It's correct whenever there's a data dependency — get_user → get_orders → summarize must be sequential.

Key Takeaways

One turn, many calls: a single assistant message can hold multiple tool_use blocks — Claude 4 batches independent calls by default.
Execution is your choice: run independent calls concurrently (asyncio.gather/Promise.all) so wall-clock latency is the critical path, not the sum (~1.8×–3.7× speedups; LLMCompiler 3.7×/6×).
Golden rule: one tool_result per tool_use, all in ONE user message (failed/skipped → is_error: true). Splitting them teaches the model to stop parallelizing — the #1 silent bug.
Sequential = data dependency: a call that needs another's output must wait; the longest dependency chain sets the floor. The model chains across turns (ReAct, L111 — Building a ReAct Agent From Scratch) and batches when independent.
Don't parallelize side-effecting, ordered, non-idempotent, or rate-limited calls — reads parallel, writes serial; when unsure, serialize.
Encourage it: the <use_parallel_tool_calls> system prompt, request phrasing, and measure avg tools/message (> 1.0). disable_parallel_tool_use: true forces one-at-a-time.
Frontier: DAG planners (LLMCompiler), programmatic tool calling (L124's sandbox), and speculative execution push latency lower still.
Next — L124: Code Execution & Sandboxing (giving the agent a safe place to run code as a tool).