Agentic Search and Multi-Hop Retrieval in Production: Query Decomposition, Evidence-Gap Loops, and Synthesis Architecture

Standard retrieval-augmented generation (RAG) relies on a single-pass paradigm: an incoming prompt is embedded, matched against a vector or hybrid index, and the top-k passages are injected into the generator's context window. This linear workflow functions reliably for direct fact retrieval, but it breaks down on complex research tasks. When a query requires chained dependencies, comparative analysis across isolated documents, or multi-step reasoning, single-shot retrieval fails because the required facts do not co-occur in a single passage or share a single dense representation.

To solve this, production retrieval architectures are transitioning from static retrieval pipelines to agentic search systems. By decoupling information gathering into iterative state machines, these architectures execute query decomposition, dynamic multi-hop traversal, evidence-gap analysis, and grounded synthesis.

The Multi-Hop Bottleneck in Static RAG

The failure mode of standard RAG on multi-hop questions stems from the representation limits of single embeddings. Benchmarks such as MultiHop-RAG demonstrate that when a question links multiple entities across disconnected documents, embedding the entire question produces an averaged vector that struggles to surface the intermediate "bridge" documents needed to unlock subsequent facts.

Consider a query such as: "Compare the 2025 data center capital expenditures of the cloud provider whose CEO sits on the board of OpenAI with the cloud provider partnering with Anthropic."

Answering this query requires four distinct information steps:

1. Identifying which cloud provider's CEO sits on the board of OpenAI.
2. Identifying which cloud provider maintains the primary compute partnership with Anthropic.
3. Retrieving the 2025 data center capex numbers for Company A.
4. Retrieving the 2025 data center capex numbers for Company B.

In a single-pass retrieval setup, the initial vector search returns general articles mentioning OpenAI, Anthropic, and cloud infrastructure, while missing specific capex filings and board seat disclosures. Agentic search reframes this task as an orchestration of discrete, dependency-aware retrieval actions.

Decomposition Architectures: Static Planning vs. Dynamic Traversal

Production systems implement query decomposition through two primary architectural patterns, often combined into a unified directed acyclic graph (DAG):

1. Static Planning and Map-Reduce Fan-Out

Static decomposition analyzes the root query once before issuing search requests. An LLM planner inspects the user intent and decomposes it into independent sub-questions that can execute concurrently.

This pattern is exemplified by systems like Stanford's STORM (Synthesis of Topic Outlines through Retrieval and Multi-perspective Question Asking). STORM addresses broad research queries by generating diverse perspective-driven sub-queries, retrieving evidence for each in parallel, and compiling the results into a structured outline before drafting sections. In production analytics, static decomposition serves queries where sub-tasks have no data dependencies on one another, minimizing end-to-end latency through parallel HTTP execution.

2. Dynamic Sequential Traversal (ReAct Loops)

When sub-queries depend on unknown intermediate values, static decomposition fails. Dynamic traversal implements an interleaved reasoning-and-acting loop based on the ReAct framework.

In dynamic multi-hop search:

• The orchestrator executes the first retrieval hop (identifying the board member and entity).
• The intermediate result is parsed into a structured entity variable.
• The next search query is dynamically constructed using the newly discovered entity.
• The loop continues until all dependency branches in the execution plan resolve.

The Evidence-Gap Critic and Corrective Loops

Retrieving documents does not guarantee that sufficient or accurate evidence has been found. Without validation, agents suffer from "retrieval hallucination," synthesizing confident answers from irrelevant or contradictory snippets. Production agentic search incorporates an explicit evidence evaluation step inspired by Self-RAG and Corrective RAG (CRAG).

After each retrieval action, an evidence critic evaluates the retrieved candidates against three operational criteria:

• Relevance Confidence: Does the snippet directly address the sub-question? CRAG classifies retrieval confidence into correct, ambiguous, or incorrect bands. Correct documents pass to context; ambiguous documents trigger secondary query reformulation; incorrect retrievals trigger fallback searches or domain-specific web indexing.
• Sufficiency and Evidence-Gap Detection: Has the specific data point (e.g., exact financial figure, date, or causal link) been extracted? If an information gap remains, the critic generates a refined query targeting the missing delta.
• Contradiction Resolution: If multiple retrieved sources present conflicting metrics, the critic flags the discrepancy and issues verification queries targeting authoritative primary sources.

Empirical studies on agentic RAG components (arXiv:2606.21553, arXiv:2606.05658) reveal a nuanced trade-off: query decomposition consistently increases topic coverage and citation accuracy across structured domains, but unconstrained reflection loops introduce substantial latency costs and can degrade ranking precision on dense multi-hop graphs if the context window is overwhelmed with noisy intermediate passages.

Engineering Safeguards for Production Deployment

Deploying iterative search agents into production environments requires strict operational bounds to avoid performance degradation and uncontrolled API spend:

1. Hard Iteration Caps and Anti-Loop Circuit Breakers

Fact-seeking agents can enter infinite search loops when attempting to verify obscure or non-existent facts. In late retrieval steps, models often repeat minor lexical variants of failed queries. Production systems enforce:

• Maximum hop limits (typically 3 to 5 iterations).
• Maximum total search tool calls (typically 8 to 15 per root query).
• Query similarity hashing: rejecting newly proposed sub-queries that exceed a 0.85 cosine similarity threshold against previously executed queries.

2. Context Distraction Mitigation via Snippet Extraction

Passing entire scraped HTML pages or dozens of raw passage chunks into the generator model degrades reasoning performance due to the distractor effect in long contexts. Efficient architectures place a lightweight extraction step (using a fast 1B to 3B parameter model or cross-encoder) between retrieval and working memory. The extractor distills 2,000-word raw scrapes into 150-word structured "evidence cards" containing only relevant factual claims and source metadata.

3. Provenance and Citation Tracking

To maintain auditability, every evidence card retains an immutable provenance envelope (source URL, content hash, retrieval timestamp, and sub-query ID). The synthesis generator is constrained to reference only verified evidence card IDs, ensuring that generated citations trace directly to inspected documents rather than hallucinations from pre-training memory.

Architectural Trade-Offs

• Naive Single-Shot RAG: Low latency (under 1.5 seconds) and minimal token cost, but fails on multi-hop reasoning and provides low evidence recall on complex topics.
• Static Map-Reduce RAG: Moderate latency (2 to 4 seconds) with high broad coverage across independent sub-topics, though unable to handle sequential data dependencies.
• Dynamic ReAct Search: Higher latency (5 to 15 seconds) and increased token spend, but successfully resolves chained dependencies and targeted depth queries.
• CRAG and Self-Reflective Agents: High and variable latency (4 to 20 seconds) with very high token overhead, providing maximum factual reliability and automated gap correction.

Agentic search replaces brittle single-shot lookups with systematic information gathering. By combining structured query planning, bounded iterative retrieval, and strict evidence verification, teams can deploy LLM systems capable of resolving complex multi-source inquiries with high factual reliability.

Sources

• MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries (Tang & Yang, 2024)
• Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models (STORM) (Shao et al., 2024)
• ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022)
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (Asai et al., 2023)
• Corrective Retrieval Augmented Generation (CRAG) (Yan et al., 2024)
• Dissecting Agentic RAG: A Component Ablation for Multi-Hop QA (arXiv:2606.21553)
• Agent-Orchestrated Adaptive RAG: A Comparative Study on Structured and Multi-Hop Retrieval (arXiv:2606.05658)