Agentic RAG

What is Agentic RAG?

Agentic RAG is retrieval-augmented generation where an agent, not a fixed pipeline, controls the retrieval process. The agent plans which sources to query, executes the searches, evaluates the results, and retries when the results fall short. Standard RAG systems follow a fixed retrieve-then-generate sequence. Agentic RAG uses a large language model to decide, at runtime, what to retrieve and whether to retrieve again.

An Agentic RAG agent has access to a set of retrieval tools: vector databases, knowledge graphs, SQL databases, and web search APIs. When it receives a query, it does not run a semantic search immediately. It first analyzes the query, breaks it into sub-tasks, selects the tools it needs, and gathers information step by step. If the retrieved context is insufficient, irrelevant, or contradictory, the agent can detect the gap, rewrite its search query, and try again before generating a response.

This loop of planning, retrieving, evaluating, and iterating lets the system handle multi-hop questions that break a fixed RAG pipeline. Deferring control-flow decisions to the LLM at runtime helps on queries that need several retrieval steps, which matters most in enterprise settings where missing context produces hallucinations.

The most visible production examples are the deep research products: OpenAI Deep Research, Gemini Deep Research, and Claude's Research mode all run an agentic retrieval loop over web search for minutes at a time, issuing dozens of queries and revising their plan as evidence accumulates. Coding agents such as Claude Code work the same way over a repository, using grep and file reads as retrieval tools instead of a vector index.

The Evolution: Naive RAG vs. Advanced RAG vs. Agentic RAG

Retrieval-augmented generation architectures fall into three stages.

Naive RAG chunks documents, embeds them, stores them in a vector database, and runs a k-Nearest Neighbors (k-NN) search on the user's raw query. The retrieved chunks go straight into the LLM's context window. This struggles with complex queries, poorly phrased questions, and large document corpora, where irrelevant chunks crowd out relevant ones.

Advanced RAG adds pre-retrieval and post-retrieval steps. Query expansion, hypothetical document embeddings (HyDE), hybrid search (dense vector plus sparse keyword search), and post-retrieval re-ranking with cross-encoders improve context relevance. The execution flow still stays fixed: the system applies these techniques in a set order, regardless of the query.

Agentic RAG removes that fixed order. The LLM decides whether retrieval is necessary, which sources to query, what parameters to pass to those sources, and whether the retrieved information is sufficient to answer the prompt. If the agent detects missing information, it starts another retrieval loop.

Key Agentic Retrieval Patterns

Agentic RAG relies on a handful of retrieval patterns that let the agent navigate multiple information sources.

Query Routing

Query routing directs a user's question to the data source or retrieval pipeline suited to it. An agent evaluates the intent of the query and routes it accordingly. A query asking "What were our Q3 revenue numbers?" might go to a Text-to-SQL agent querying a relational database, while "What is our policy on remote work?" would go to a vector search over the employee handbook. Routing reduces noise by matching the query to the tool built to answer it.

Query Decomposition

Many questions are too complex for a single search query. Query decomposition (or sub-querying) is how an agent breaks a multi-part question into smaller, independent search tasks. Asked "How did our profit margins in 2023 compare to the industry average?", the agent might decompose this into two queries:

1. "Retrieve company profit margins for 2023."
2. "Retrieve industry average profit margins for 2023." The agent runs these queries in parallel or in sequence, then combines the retrieved contexts into the final comparison.

Self-RAG

Self-Reflective Retrieval-Augmented Generation (Self-RAG) is a framework where the agent evaluates its own retrieval and generation. The model is trained or prompted to generate reflection tokens: first, "Do I need to retrieve information to answer this?" After retrieval, "Is this retrieved context relevant to the query?" During generation, "Is my generated response fully supported by the retrieved context?" This lets the agent discard irrelevant information and reduces hallucinations.

Corrective RAG (CRAG)

Corrective RAG is a fallback mechanism inside the agentic loop. When an agent retrieves documents, a lightweight evaluator, often another LLM prompt or a smaller model, grades their relevance. If the confidence score is too low, CRAG triggers a corrective action: rewriting the query, broadening the search parameters, or falling back to a web search to find the missing context. It gives the agent a check against confidently answering from poor retrieval results.

Control Flow and Multi-Step Execution Loop

The core of an Agentic RAG system is a multi-step execution loop, usually implemented as a state machine or a directed acyclic graph (DAG) using frameworks like LangGraph, LlamaIndex Workflows, or custom orchestration logic.

A typical execution loop proceeds through the following states:

1. Input Reception & Intent Classification: The agent receives the user query and determines the overall goal.
2. Planning & Decomposition: The agent creates an execution plan, breaking complex queries into smaller steps.
3. Tool Selection: Based on the current step in the plan, the agent selects the appropriate retrieval tool (e.g., vector DB, SQL, API).
4. Action Execution (Retrieval): The agent formulates the specific query parameters and executes the tool.
5. Observation & Context Evaluation: The agent receives the retrieved data and checks it for relevance, completeness, and factual consistency.
6. State Update & Decision (The Loop):
   • If the information is sufficient, the agent moves to the generation phase.
   • If the information is insufficient, the agent updates its internal state (memory of what it has tried), reformulates the query, selects a different tool, or moves to the next sub-task, returning to step 3.
7. Synthesis & Generation: The agent combines the gathered, verified context into a final response.

This cyclical process requires the agent to keep state, a scratchpad or memory of its intermediate thoughts, tool calls, and observations, so it does not get stuck repeating the same failed query.

How Much Does the Loop Actually Help?

The clearest public evidence comes from OpenAI's BrowseComp benchmark (April 2025), 1,266 questions built so that the answer cannot be found with a single search. GPT-4o with a plain browsing tool scored 1.9%, while agentic systems trained to plan, search repeatedly, and verify reached roughly 50-78%. The gap is the value of the loop itself: the questions are unanswerable without iterative retrieval.

BrowseComp-Plus (August 2025, later an ACL 2026 paper) re-ran 830 of these queries against a fixed corpus of about 100K human-verified documents so the retriever and the agent could be varied independently. Two results are worth knowing. First, the agent model dominates: GPT-5 scored 55.9% where a small open-source search agent scored under 4% with the same BM25 retriever. Second, the retriever still matters even inside an agentic loop: swapping BM25 for a dense Qwen3-Embedding-8B retriever lifted GPT-5 to 70.1% while reducing the number of search calls it needed. An agent can compensate for a weak retriever by searching more, but it pays for that in latency and tokens.

The practical reading: agentic control flow is what makes hard multi-hop questions answerable at all, but investing in retrieval quality (hybrid search, reranking) still pays off, because it shortens the loop.

Cost, Latency Budgets, and Token Compounding in Agent Loops

Agentic RAG improves accuracy on complex tasks at a cost: it introduces real latency and spend to manage.

Traditional RAG involves a single prompt to an LLM. Agentic RAG involves multiple LLM calls within one user interaction. Every time the agent plans, evaluates relevance, or decides on the next step, it consumes tokens. This is called token compounding.

Because the input context (the system prompt, tool descriptions, and accumulated conversation history or scratchpad) is passed to the LLM on every iteration of the loop, the token count grows fast. This leads to:

• Increased latency: Sequential LLM calls take time. A loop that runs three or four iterations can push response times to 10-20 seconds or more, which is too slow for synchronous, user-facing chat.
• Higher costs: Paying for input tokens processed repeatedly across multiple steps raises the cost per query compared to naive RAG pipelines.

To manage this, engineers set explicit latency and cost budgets:

• A hard limit on the maximum number of iterations (e.g., max_steps = 5).
• Smaller, faster models, such as Claude Haiku 4.5, GPT-5 mini, or Gemini 3 Flash, for intermediate routing and evaluation steps, reserving larger models like GPT-5 or Claude Opus 4.5 for the final synthesis.
• Semantic caching to bypass the agent loop for frequently asked questions.

Evaluating and Monitoring Agentic Retrieval Pipelines

Evaluating Agentic RAG is more complex than evaluating a standard generative model, since it requires assessing both the intermediate steps and the final output. The industry standard framework (similar to RAGAS) uses a set of metrics to check the health of the pipeline.

Context precision measures the signal-to-noise ratio of the retrieved documents. When the agent queries a database, are the top-ranked results relevant, or is the agent retrieving irrelevant content it then has to filter out? High context precision means the agent's tool calls are well formed.

Context recall measures whether the agent retrieved all the information the query needs. If a question requires three pieces of information and the agent retrieves only two before generating a response, context recall is low. In an agentic system, low recall usually points to a failure in query decomposition or routing.

Hallucination rate (faithfulness) measures whether the final answer is derived from the retrieved context. An agent can retrieve the right information and still hallucinate during synthesis, relying too heavily on its pre-trained parametric knowledge instead of the supplied facts. LLM-as-a-judge methods are commonly used to score this automatically.

Monitoring these systems in production requires tracing frameworks, such as LangSmith, Phoenix, or Klu's internal tracing, to log every thought, tool call, and observation. Reviewing these traces is how teams find loops where the agent gets stuck, fails to use a tool correctly, or misreads the retrieved data.

Frequently Asked Questions

How does Agentic RAG differ from standard function calling? Function calling is a feature of the LLM that lets it output structured data matching a tool's schema. Agentic RAG uses function calling to interact with retrieval systems, but the architecture is broader: it is the autonomous loop of planning, executing those functions, evaluating the output, and calling more functions until the query is answered.

Can I use Agentic RAG for real-time customer support? Yes, but latency is the limiting factor. Agentic loops require multiple sequential LLM calls, so time-to-first-token is often high. For real-time applications, use fast models, cap the number of agent steps, and stream the agent's intermediate reasoning to the user so they can see it working.

Do I need a graph database for Agentic RAG? No. Agentic RAG can operate over standard vector databases, SQL databases, and external APIs. Knowledge graphs (GraphRAG) pair well with agentic systems because they let the agent traverse relationships between entities directly, which helps with multi-hop reasoning tasks.

Is agentic search replacing vector-based RAG? Partially. For codebases and small corpora, agents that grep and read files directly often beat an embedding index, because the agent can follow exact identifiers and iterate. For large document collections, a search index still matters: on BrowseComp-Plus, the same GPT-5 agent scored 14 points higher with a dense retriever than with BM25. Most 2026 production systems give the agent both a keyword tool and a semantic search tool and let it choose.

What happens if the agent gets stuck in an infinite loop? Without safeguards, an agent can repeatedly fail to retrieve the right information and keep retrying the same failed query. Production systems handle this with iteration caps (e.g., stopping after 5 attempts) and a system prompt instruction to fall back to a plain "I could not find the answer" response if the retrieval strategy fails.