The RAG Quality Problem Nobody Talks About: Context Fragmentation

Your RAG system works well until the second question.

1User: "Show me Q4 revenue for Acme Corp"
2AI:   "Acme Corp Q4 2025 revenue was $12.4M, up 23% YoY."
3
4User: "How does that compare to their competitors?"
5AI:   "I'd be happy to help compare revenues. Which company
6       would you like me to analyze?"   ← context lost

The user was obviously asking about Acme Corp's competitors. Nothing errored. The retrieval simply ran on a question that no longer contained the thing being asked about.

If your architecture splits conversation history across several agents, this failure is not a bug you introduced. It is a property of the shape.

What the numbers looked like

We pulled 5,000 RAG queries from our own multi-agent system in February 2026 and scored them by question type rather than in aggregate, which is the only reason the pattern was visible at all:

First questions were fine. Follow-ups failed at close to three times the rate.

A single blended accuracy number would have shown roughly 80% and told us nothing. The split is what pointed at the cause.

Why the query arrives incomplete

Walk a two-turn conversation through a router-and-specialists architecture:

1Turn 1: User asks about Acme Corp
2        │
3        ▼
4┌──────────────────────────────────────┐
5│ ROUTER: sends to the General Agent   │
6│ → General Agent queries RAG          │
7│ → Retrieves: "Acme Corp Q4 revenue"  │
8│ → Response generated                 │
9└──────────────────────────────────────┘
10        │
11        │  (context so far: an Acme Corp revenue discussion)
12        │
13Turn 2: User asks about competitors
14        │
15        ▼
16┌──────────────────────────────────────┐
17│ ROUTER: reads "compare to their      │
18│         competitors"                 │
19│                                      │
20│ The router sees THIS message only    │
21│ No access to turn 1                  │
22│                                      │
23│ → Routes to the Research Agent       │
24│ → Research Agent has no context      │
25│ → Generates a vague query            │
26└──────────────────────────────────────┘

Every component is holding a fragment. The router has the current message and perhaps a summary. The specialist has its own tools and a truncated history. The query generator has whatever reached it.

So the embedding search runs on compare competitors when the question was Acme Corp competitors Q4 revenue. The index does its job perfectly and returns the wrong thing, because the wrong thing is what it was asked for.

This is well described outside our own logs too. The multi-agent literature calls it the disconnected models problem: contextual information distributed across agents with no single agent holding a complete picture, producing inconsistent understanding downstream.

The fix is architectural, not a prompt

There is no retrieval tuning that recovers a word the query never contained. The change has to happen before the search, in what the agent is holding when it writes the query. Curating that window is the same discipline as deciding which tokens are worth re-sending on every turn.

Fragmented, which is where we started:

1# The router only sees the current message
2router_context = [current_message]  # no history
3
4# So the RAG query is built without it
5rag_query = generate_query(current_message)
6# Returns: "compare competitors"  ← too vague to retrieve on

Unified, which is where we ended up:

1# One agent holds the whole conversation
2agent_context = [
3    *message_history[-10:],
4    current_message
5]
6
7# The query is generated from all of it
8rag_query = generate_query(agent_context)
9# Returns: "Acme Corp competitors Q4 revenue comparison"

Structurally the move is from a chain of partial views to a single one:

1Fragmented:  Router → Agent A → Agent B   (three partial windows)
2
3Unified:     One agent, full history, skills activated as needed

The capabilities do not shrink. What changes is that specialisation becomes a set of tools the one agent can load, rather than a set of colleagues each holding a different piece of the conversation. The mechanics of loading capability without drowning the agent in it are a separate problem, and giving an agent too many tools has its own accuracy curve.

What consolidating was worth

The last row is the one that says the most. Conversations got longer because they stopped breaking. People asked a second and third question because the second and third question worked.

The measurement worth stealing

If you take one thing from this, take the diagnostic rather than the fix: score your retrieval separately for first questions, follow-ups, and references to earlier turns. It costs almost nothing and it is the difference between "our RAG is about 80%" and knowing exactly which 20% is failing and why.

For anything conversational, which is to say anything people actually use, unified context is not a refinement. It is the thing that makes the second question work.

FAQ

What is context fragmentation in a RAG system?
Conversation history spread across components with no single one holding all of it, so the search query gets built from a fragment and retrieves against the wrong question.

Why do follow-up questions fail more often than first questions?
First questions are self-contained. Follow-ups depend on the turn before them, and if the query generator cannot see that turn it embeds the vague version.

How do you fix it?
One agent holding the whole conversation, generating the retrieval query from all of it, with specialisation loaded as skills rather than delegated to agents with partial history.

How do you tell whether your system has this problem?
Score first questions, follow-ups and back-references separately. A blended number hides the failure entirely.