Why Your AI Router Is Your Biggest Performance Bottleneck

The routing agent is the most reasonable-looking component in a multi-agent system. A supervisor reads the query, decides whether it is research or code or general, and hands it to the right specialist. Modular, legible, easy to draw on a whiteboard.

It also runs a language model call on every single request, and produces nothing the user ever sees.

1User: "What's my account plan?"
2    │
3    ▼
4┌──────────────┐
5│   ROUTER     │  ← LLM call #1 (300ms)
6│  decides:    │
7│  "this is    │
8│   accounts"  │
9└──────┬───────┘
10       │
11       ▼
12┌──────────────┐
13│   GENERAL    │  ← LLM call #2 (700ms)
14│    AGENT     │
15└──────────────┘
16
17Total: 1,000ms for a simple question

Seven hundred milliseconds of that produced an answer. Three hundred produced a decision about who should produce the answer.

The tax, measured

We pulled 10,000 production queries in February 2026 and compared the routing layer against what we had assumed when we built it:

Every row was worse than the estimate, and each for a mundane reason.

The latency estimate priced a small model call and forgot everything around it. The real call carries a routing system prompt of about 400 tokens, roughly 300 tokens of conversation history, because a router with no history routes badly, and about 200 tokens of response to parse. That is a real request, not a lookup.

The token number is higher than expected because context gets paid for twice. The router needs the conversation to decide. The specialist needs the same conversation to answer. Nothing about the second call is cheaper for the first having happened.

And the accuracy row compounds the other two. At 87%, roughly one query in eight routes somewhere unhelpful, and correcting that costs another full round trip.

At 100,000 queries a month, the routing layer alone came to about $0.002 a query, or $200 a month spent deciding who should do the work.

What the guidance actually says

It is worth being precise here, because this area attracts confident-sounding claims.

Anthropic's published guidance on building effective agents recommends finding the simplest thing that works and adding complexity only when it demonstrably helps, and is explicit that agentic systems trade latency and cost for task performance. The framework documentation from the major vendors converges on the same practical point: orchestration earns its cost on work that genuinely fans out, and does not on work that runs in sequence.

Most product traffic runs in sequence. A user asks something, the agent looks something up, the agent answers. There is nothing to parallelise and therefore nothing for a router to buy you.

Activating instead of delegating

The alternative is not a worse-organised system. It is one agent that reaches for capabilities rather than handing off to colleagues:

1User: "What's my account plan?"
2    │
3    ▼
4┌─────────────────────────────────────┐
5│          UNIFIED AGENT              │  ← LLM call #1 (700ms)
6│                                     │
7│  Core: company tools, always ready  │
8│                                     │
9│  Skills, loaded when needed:        │
10│  ├── analytics                      │
11│  ├── documents                      │
12│  ├── research                       │
13│  └── code                           │
14│                                     │
15│  Recognises "account" → uses core   │
16│  Responds directly                  │
17└─────────────────────────────────────┘
18
19Total: 700ms

The classification work has not disappeared. It happens inside the call that also produces the answer, rather than in a separate round trip whose entire output is a routing label. Keeping that tool list short enough for the agent to choose well is what makes this work at all, and agent tool overload is the failure it prevents.

What removing the router changed

The routing errors row is the one that surprised us. We had treated 13% re-routes as a tuning problem, something a better router prompt would fix. Removing the router removed the error class entirely, because a decision you never make cannot be made wrong.

Where routing still earns its keep

This is not an argument that multi-agent architectures are a mistake. Two cases still justify the overhead.

Genuinely parallel work. Auditing twelve competitors or surveying a literature is work that fans out cleanly, and real concurrency beats a single sequential agent by more than the coordination costs.

Isolation you actually need. Keeping certain tools or data strictly separate is a security requirement, and paying latency for it is a reasonable trade.

Neither describes a user asking a question and waiting for an answer. If your agents run one after another anyway, the orchestration is buying you a diagram rather than a capability.

The supervisor pattern this router came from was a real fix for a real problem, and the wider case against going multi-agent is what we landed on after measuring what it cost. On the product side it is why one agent replaces a stack of tools rather than coordinating a team of them.

FAQ

How much latency does a routing agent actually add?
Ours averaged 300ms across 10,000 queries, against a 100ms estimate. The estimate usually omits the routing prompt, the history the router needs, and the parse.

Why is routing token overhead higher than expected?
The conversation is paid for twice, once so the router can decide and once so the specialist can answer.

When is a router still the right call?
Genuinely parallel work, and isolation you need for security reasons. Not sequential request-and-answer traffic.

What replaces routing?
One agent that activates capabilities rather than delegating, so the decision happens inside the call that produces the answer.