How We Built a Multi-Agent AI System That Routes 10x Faster

First written in early 2026, when Savra ran a multi-agent architecture. This is the most complete description of that system we published, and we have since taken most of it apart: we now run a single agent with dynamically activated skills, for the reasons in Don't Go Multi-Agent and Why Your AI Router Is Your Biggest Performance Bottleneck. The token work below survived the change. The orchestration around it did not. Both halves are worth seeing in one place.

Users wanted to do everything through one chat box:

• "Search the web for competitor analysis"
• "What products does my company offer?"
• "Help me debug this Python function"
• "Find my uploaded documents about marketing"

One agent with 25 tools attached handled all of it badly, for the reasons tool overload makes measurable. Eight thousand tokens went into describing the tools before the user's question was even considered, which meant one or two requests a minute against our rate limits, and the model regularly picked the wrong tool because it had twenty-five plausible ones to choose from.

The shape we built

1                         ┌─────────────────┐
2                         │   SUPERVISOR    │
3                         │  (router agent) │
4                         │    0 tools      │
5                         └────────┬────────┘
6                                  │
7              ┌───────────────────┼───────────────────┐
8              ▼                   ▼                   ▼
9     ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
10     │    GENERAL      │ │    RESEARCH     │ │      CODE       │
11     │  Company data   │ │   Web search    │ │  Programming    │
12     │    14 tools     │ │    10 tools     │ │     4 tools     │
13     └─────────────────┘ └─────────────────┘ └─────────────────┘

Following one request through

Take "what does this company do?" about an organisation we hold no data on.

The supervisor decides, and only decides. Its entire output is a typed object:

1class RoutingDecision(BaseModel):
2    agent: Literal["research", "code", "general"]
3    reason: str

1{
2  "agent": "research",
3  "reason": "User asking about external company - requires web search"
4}

About two seconds and roughly 100 tokens, because no tools are loaded to describe.

The specialist loads with its own tools:

1agent_tools = get_tools_for_agent("research")
2# Returns: [web_search, web_search_news, crawl_url,
3#           extract_page_summary, search_knowledge_base, ...]

It decides to crawl the company's about page, which takes fifteen seconds of wall-clock time that no architecture will give back:

119:10:33 → Tool call:   crawl_url("https://example.com/about")
219:10:48 → Tool result: [full page content extracted]

Then it synthesises and streams, saving to Postgres for history and logging to Langfuse for observability. Around sixteen seconds end to end, nearly all of it the crawl.

The token work, which is the part that mattered

Every agent used to load every tool:

1# Old approach: every agent got all 25 tools
2tools = get_all_tools()  # 8,000+ tokens in prompts

The fix is a dictionary:

1# New approach: dynamic per-agent loading
2AGENT_TOOLS = {
3    "router": [],                    # 0 tools
4    "general": [...14 tools...],     # company data, memory
5    "research": [...10 tools...],    # web search, crawling
6    "code": [...4 tools...],         # KB search, docs
7}
8
9def get_tools_for_agent(agent_type: str) -> list:
10    return AGENT_TOOLS.get(agent_type, [])

The per-agent cost, once loading is dynamic:

Against roughly 8,000 tokens when every agent carried the full set.

Docstrings are a token budget

The second saving came from writing tool descriptions like they cost money, because they do:

1# Before (~150 tokens)
2"""
3Get all products for the current company.
4
5Use this tool when the user asks about:
6- What products are available
7- Product catalog or offerings
8- Product pricing or features
9
10Args:
11    active_only: If True, only return active products
12
13Returns:
14    Formatted list of products with name, price, and features
15"""
16
17# After (~40 tokens)
18"""Get company products (name, price, features). Use for product queries."""

A good tool name plus one line does the job. The structured block was written for a human reader who was never going to read it, and paid for on every single request.

Building for the failures you know are coming

Rate limits are not an edge case, so we stopped treating them as one:

1from pydantic_ai.models.fallback import FallbackModel
2
3def get_model_with_fallback(primary_model: str) -> FallbackModel:
4    """
5    Primary: Claude Sonnet
6    Fallback: Gemini Flash (on rate limit)
7    """
8    primary = AnthropicModel("claude-sonnet-4-5")
9    fallback = GeminiModel("gemini-3-flash-preview")
10
11    return FallbackModel(primary, fallback)

Five lines that convert a visible outage into a quality difference most users never notice.

The stack underneath all of it:

Making the system legible

Every node logs on entry and exit, with structure rather than prose:

1logger.info(
2    "RESEARCH AGENT: Starting",
3    tool_count=len(agent_tools),
4    tools=[t.__name__ for t in agent_tools],
5)
6
7# ... agent execution ...
8
9elapsed = time.time() - start_time
10logger.info(
11    "RESEARCH AGENT: Complete",
12    elapsed_ms=round(elapsed * 1000, 2),
13    tokens_used=usage_data.get("total", 0),
14    response_length=len(result.output),
15)

Which reads back as a timeline you can reason about:

1SUPERVISOR: Starting routing decision | mode=routing | message_count=1
2SUPERVISOR: Routing complete | next_agent=research | elapsed_ms=1823.45
3RESEARCH AGENT: Starting | tool_count=10 | tools=['web_search', 'crawl_url', ...]
4RESEARCH AGENT: Complete | elapsed_ms=14532.12 | tokens_used=28924 | response_length=2847

Two numbers in that trace are worth sitting with. The routing decision cost 1.8 seconds and produced no output the user sees. The research agent consumed 28,924 tokens against the general agent's 2,300 for a comparable turn.

Both are the sort of thing you only notice once it is logged, which is the argument for tracing every agent run before you optimise anything. Both eventually became reasons to change the architecture.

Streaming the reasoning

Users tolerate a sixteen-second answer far better when they can watch it happen:

1{ "event": "thinking",
2  "data": { "thinking": "Analyzing your request..." } }
3{ "event": "routing",
4  "data": { "routing": { "agent": "research", "reason": "Web search needed" } } }
5{ "event": "tool_call",
6  "data": { "tool": "crawl_url", "args": { "url": "..." }, "status": "started" } }
7{ "event": "text",
8  "data": { "text": "This company is a business transformation..." } }
9{ "event": "done" }

This is the piece we would build again unchanged. Showing the work turns waiting into watching, and it turns a black box into something a user can develop an accurate mental model of.

What held and what did not

Separation is real, and it was not the agents. Not giving every agent every tool is what produced the 90% saving. That is dynamic loading, and it works just as well with one agent as with four.

Models do not need verbose documentation. Names and one-line summaries, every time.

Build for failure. Fallback models make rate limits invisible.

Stream everything. Visible reasoning buys patience and trust.

The supervisor is the part that did not survive. It costs a model call on every request to produce a routing label, it needs conversation history to route well so the same context is paid for twice, and splitting that history across specialists quietly degrades retrieval the moment a user asks a follow-up question. We measured each of those afterwards, and each has its own article.

If you are building this today, take the tool registry and the fallback model and the streaming. Read Don't Go Multi-Agent before you build the supervisor.

FAQ

What does a supervisor agent actually do?
Reads the message, returns one structured decision naming the specialist. Holding no tools, it costs around 100 tokens.

How much do verbose tool docstrings cost?
Roughly 150 tokens each against about 40 for a one-line version, paid on every request.

How do you handle provider rate limits in production?
A fallback model. Requests move to a secondary provider automatically when the primary refuses.

Is this architecture still what you would build?
No. The dynamic tool loading survived; the supervisor did not.