Building an Agent Hub with Full Observability

First written in February 2026, when Savra ran a multi-agent architecture, so the hub below is a directory of separate agents. We have since moved to a single agent with dynamically activated skills, and the reasons are in Don't Go Multi-Agent. The observability half of this article survived the change entirely: the tracing described here is what made the case for that change measurable.

As the system grew, two different people had the same complaint in different words. Users could not tell which agent they should be talking to. We could not tell which agent was costing us money.

Both problems came from the same gap. Nothing described the agents in one place, and nothing recorded what they did.

Agents as data, not code

The registry is one config object per agent:

1@dataclass
2class AgentConfig:
3    name: str              # "research"
4    display_name: str      # "Research Assistant"
5    description: str       # for the hub UI
6    icon: str              # "🔍"
7    category: str          # "Research"
8    qa_pattern: QAPattern  # QAPattern.SELF_PLUS_SUPERVISOR
9    enabled: bool          # show in the hub?
10    tools: list            # dynamically loaded

Everything that lists agents reads from it:

1@router.get("/api/v1/agents")
2async def list_agents():
3    return {
4        "agents": [
5            {
6                "id": cfg.name,
7                "display_name": cfg.display_name,
8                "description": cfg.description,
9                "icon": cfg.icon,
10                "category": cfg.category,
11                "is_active": cfg.enabled,
12                "qa_pattern": cfg.qa_pattern.value,
13                "tools": get_tools_for_agent(cfg.name),
14            }
15            for cfg in AGENT_CONFIGS.values()
16        ]
17    }

Adding an agent to the registry makes it appear in the hub. There is no second place to update, which matters more than it sounds, because the alternative is three descriptions of the same agent drifting apart over a fortnight.

The hub itself is a card per agent with its icon, category, description and quality pattern, and a profile page behind it carrying an overview, its capabilities and some example prompts. Telling users which quality pattern an agent runs turned out to be worth more than we expected: people trust a slow answer more when they know a review step is why it is slow.

What tracing actually gives you

Every conversation produces a trace, and each trace carries the whole run:

1. A trace ID for the conversation
2. One span per step in the workflow
3. Every tool call, with its arguments and its result
4. Token counts, split into input and output
5. Cost per call, per model

Here is a real run, a supervisor routing a question to a specialist which then fans out across four tools:

1general run
2├── chat claude-sonnet-4-5           2.06s   2,778 → 76   $0.009474
3├── running 1 tool
4│   └── search_knowledge_base        1.32s
5└── running 3 tools
6    ├── get_products
7    ├── get_services
8    └── get_company_info

That one block answers questions that are otherwise guesswork. The first model call took 2.06 seconds and cost $0.009474 for 2,778 input tokens against 76 output. The knowledge base search took 1.32 seconds, which is more than half the perceived latency of the whole turn and the obvious place to look first.

The integration is unobtrusive:

1from langfuse import Langfuse
2
3# Initialise once
4langfuse = Langfuse()
5
6# Every agent call is traced automatically
7@trace_agent(name="research")
8async def research_agent_node(state: GraphState):
9    # ... agent logic ...

Why this comes first

Every cost and latency win we have written about since started as a line in a trace.

We found that almost all of our input tokens were unchanged context because the trace broke input down by component, which is what made prompt caching an obvious move rather than a speculative one. We found that the routing layer cost three times what we had assumed because the router's own call was a span with a duration on it. Neither would have been visible on an invoice.

It is also how we found the event boundary that was silently dropping our state between turns: the trace showed the tool running, and the parent stream showed nothing at all.

The general form: without per-request attribution you know the bill is high and nothing about which part of the request caused it. Several of our confident hypotheses about where the money was going turned out to be wrong, and the traces were what settled it.

What you can build on top

Once every run is traced, a few things become straightforward rather than projects:

• Usage analytics. Which capabilities do people actually reach for, as opposed to which ones we assumed they would.
• Cost dashboards. Token spend per agent per day, and per customer, which is the number you need before pricing anything.
• Quality monitoring. Review rejection rates by agent, and the reasons attached.
• Performance work. Slow tools make slow agents, and the trace names the tool.

The order matters. Instrument first, then optimise, because the alternative is optimising the part you happened to think of.

FAQ

Why define agents as configuration rather than code?
So the directory, the API and the profile pages read the same facts. Three copies of those facts drift within a week.

What is worth tracing on an AI agent?
The trace, a span per step, every tool call with arguments and result, input and output token counts, and cost per call.

Why does observability come before optimisation?
Without per-request attribution you know the bill is high and nothing about why. Our confident guesses were often wrong.

What can you build once every agent run is traced?
Usage analytics, per-agent and per-customer cost dashboards, quality monitoring from rejection rates, and slow-tool detection.