3x Faster AI Responses with Parallel Tool Execution

Ask an agent to compare your revenue against two competitors and it needs three lookups. If it runs them one after another, the user waits about a second and a half while nothing happens in parallel that easily could.

Those three searches do not depend on each other. Run them at once and the wait becomes the length of the slowest one, around 500ms.

Why does a three-lookup answer take three times as long?

Here is the sequential version, which is what you get by default:

1Step 1: Search internal KB      (500ms)
2Step 2: Search competitor A     (500ms)
3Step 3: Search competitor B     (500ms)
4─────────────────────────────────────
5Total: 1,500ms waiting on I/O

Every one of those milliseconds is spent waiting on the network. The CPU is idle, the searches have nothing to say to each other, and the user is watching a spinner.

Overlap them and the arithmetic changes:

1Step 1: Search internal KB ─────────┐
2Step 2: Search competitor A ────────┼──► Results gathered
3Step 3: Search competitor B ────────┘
4─────────────────────────────────────
5Total: 500ms (the slowest one)

Same results, same order in the output, a third of the wait.

Does the framework already handle this?

Partly, and it is worth knowing which part. When the model asks for several tools in one turn, Pydantic AI schedules them concurrently on its own:

1# The model returns multiple tool calls in one turn:
2# [search_kb("our revenue"),
3#  web_search("competitor A revenue"),
4#  web_search("competitor B revenue")]
5
6# Pydantic AI schedules them as:
7asyncio.create_task(search_kb(...))
8asyncio.create_task(web_search(...))  # concurrent
9asyncio.create_task(web_search(...))  # concurrent

So the common case is already fast. The gap is the turn-by-turn case: the model decides to look one thing up, reads the result, then decides to look up the next. Nothing is running concurrently there, because from the framework's point of view each turn had exactly one tool call in it.

That is the case an explicit parallel tool exists to close.

Giving the model one tool that fans out

Instead of hoping the model batches its requests, hand it a tool whose whole job is to run a list of queries at once:

1import asyncio
2from pydantic_ai import RunContext
3
4async def parallel_search(
5    ctx: RunContext,
6    queries: list[str]
7) -> list[dict]:
8    """
9    Execute multiple searches in parallel.
10
11    Example:
12        User: "Compare products A, B, and C"
13        Agent: parallel_search(["product A", "product B", "product C"])
14    """
15
16    async def search_one(query: str) -> dict:
17        result = await search_knowledge_base(ctx, query)
18        return {"query": query, "result": result}
19
20    # Run every query simultaneously
21    results = await asyncio.gather(
22        *[search_one(q) for q in queries]
23    )
24
25    return list(results)

The docstring is doing real work here. It is what tells the model that a comparison across several things is one call rather than several turns.

Keeping it inside your rate limits

Unbounded fan-out is how you discover an upstream API's rate limit in production. A semaphore keeps the overlap and caps the width:

1async def parallel_research(
2    ctx: RunContext,
3    topics: list[str],
4    max_concurrent: int = 3
5) -> list[dict]:
6    """Research topics in parallel with rate limiting."""
7
8    semaphore = asyncio.Semaphore(max_concurrent)
9
10    async def research_one(topic: str) -> dict:
11        async with semaphore:  # only three at a time
12            return await do_research(topic)
13
14    return await asyncio.gather(
15        *[research_one(t) for t in topics]
16    )

Three concurrent calls captures most of the win. The curve flattens quickly, because the benefit comes from overlapping the first few waits rather than from running everything at once.

What it was worth

Measured across 1,000 multi-lookup queries in February 2026:

The speedup grows with the number of lookups, because the total stops being a sum and becomes the slowest call plus a little overhead. On a five-way product comparison that was better than 4x.

End to end, across all traffic rather than just the multi-lookup subset:

The P99 number is the one that mattered. A median of 2.1 seconds is survivable; a tail of 4.5 seconds is where people stop trusting that the thing is working.

The rule worth keeping

Independence is the whole test, and it is the same question that decides whether an AI marketing team can run five channels at once or has to walk them one at a time. Before parallelising anything, ask whether any call needs another's output. If the answer is no, the sequential version is pure waste, and the fix is one asyncio.gather away. If the answer is yes, leave it alone, because a race condition costs far more than the second you were trying to save.

The other half of the latency problem is the context you re-send on every turn, which we cut separately with prompt caching.

FAQ

When can an agent's tool calls run in parallel?
When none of them needs another's output. If step two consumes step one's result, it stays sequential.

Does the framework already do this?
Pydantic AI schedules concurrently when the model requests several tools in one turn. It cannot merge calls the model made across separate turns, which is what an explicit parallel tool covers.

How do you keep parallel calls from tripping rate limits?
An asyncio.Semaphore sized to the slowest upstream API. You keep the overlap and cap the width.

How much faster does this actually get?
Two lookups roughly halved, three ran about 3x faster, five better than 4x. The gain scales with how many independent calls you had.