TL;DR
- An agentic RAG pipeline treats retrieval as a tool the AI agent can call, evaluate, and call again rather than as a fixed step.
- The pipeline can search an internal knowledge base first, then use real-time web search when the available evidence is missing, weak, or outdated.
- Internal documents and web results should be converted into a shared evidence format before the model generates an answer.
- A reliable system must preserve URLs, publication dates, document identifiers, and the claims supported by each source.
- Retrieval quality, web-search precision, citation correctness, latency, cost, and stopping behaviour should all be evaluated.
A basic RAG pipeline works well until the answer is not in the knowledge base.
Imagine an enterprise copilot that can answer questions about internal product documentation. It performs semantic search against a vector database, retrieves several relevant passages, and passes them to a language model. For questions covered by the indexed documents, the system may work remarkably well.
Then a user asks about a release announced yesterday, a recently changed regulation, or how the company’s product compares with a new competitor.
The vector database cannot retrieve information it has never indexed. A conventional pipeline may return no answer, but it may also produce a confident response from incomplete or outdated context.
Adding a Web Search API helps solve the freshness problem, but it introduces another decision: when should the system trust its internal knowledge, and when should it search the open web?
An agentic RAG pipeline places that decision inside the retrieval workflow.
What Makes a RAG Pipeline Agentic?
A traditional RAG pipeline usually follows a fixed path: transform the question into a search query, retrieve the most similar passages, add those passages to the prompt, and generate an answer.
An agentic RAG pipeline allows the model to make decisions between those stages. Retrieval becomes a tool rather than a mandatory one-time operation.
The agent can determine what information the question requires, decide which source to search, inspect the evidence, reformulate the query, and retrieve again. It can also decide that the evidence is already sufficient and skip unnecessary searches.
This does not mean every stage has to be autonomous. The strongest architectures often combine deterministic controls with a limited number of model-driven decisions. Search budgets, domain restrictions, evidence schemas, and citation requirements can remain fixed even when the agent controls query planning and routing.
Microsoft’s agentic RAG architecture guidance describes a similar pattern: retrieval is exposed as a tool that an agent can invoke while reasoning across several information sources.
For a broader comparison of the two approaches, see Agentic RAG vs. Traditional RAG: How AI Agents Improve Retrieval.
The Architecture of an Agentic RAG Pipeline
A practical pipeline can begin with internal retrieval and expand to the web only when the initial evidence is inadequate.
User query
↓
Intent and query planner
↓
Internal vector search
↓
Evidence grader
├── Sufficient ─────────────→ Answer with citations
│
└── Missing, weak or stale
↓
Web Search API
↓
Extract, normalise and deduplicate
↓
Optional follow-up search
↓
Answer with citations
The planner first determines whether the question contains several information needs. A broad request may need to be divided into smaller queries before any retrieval begins.
The internal retriever then searches the existing knowledge base. This stage can use vector search, keyword search, or a hybrid approach. The important point is that the retrieved documents are not passed directly to the answer generator.
An evidence grader checks whether those documents can support the requested answer. It should assess coverage, freshness and direct support—not just embedding similarity.
If the evidence is sufficient, the pipeline can answer without using the web. If it is incomplete or outdated, the agent calls the Web Search API. The resulting pages are extracted, normalised, deduplicated and added to the same evidence store used for internal documents.
This design gives web search a clear role. It is neither a permanent first step nor an unstructured last resort. It is an external evidence source invoked under defined conditions.
Build the Pipeline Around Evidence, Not Tools
The most useful way to design this system is to begin with the evidence required by the final answer. Tools are simply different ways of obtaining that evidence.
Expose Internal Retrieval and Web Search as Separate Tools
The internal retrieval tool should return the document text together with information such as the document ID, collection, section and retrieval score. The web-search tool should return a title, URL, snippet, source, publication date and, when necessary, extracted page content.
Keeping the tools separate makes routing visible. It becomes possible to determine whether the agent searched the web unnecessarily, ignored a useful internal document, or relied on a snippet when it should have inspected the complete page.
The tools should retrieve information rather than generate final answers. If a search tool silently summarises its results, the pipeline may lose the connection between the original evidence and the claims produced later.
LangGraph’s custom RAG agent guide demonstrates this separation through dedicated retrieval, document-grading and query-rewriting stages. The same principle applies even if the pipeline uses another framework or a custom orchestrator.
Add a Router and an Evidence Grader
The router decides where the search should begin. A question about an internal policy probably belongs in the knowledge base. A question containing phrases such as “latest”, “today” or “current price” is more likely to require real-time search.
Some questions need both sources. A support agent might use internal documentation to explain how a product works, then search the web for a newly disclosed vulnerability affecting one of its dependencies.
After internal retrieval, the evidence grader determines whether the results are sufficient. It should distinguish between several failure modes: no documents were retrieved, the documents are off-topic, the information is relevant but incomplete, or the information is too old for the question.
This is where corrective RAG and adaptive RAG ideas become useful. The Corrective RAG paper proposes assessing retrieved documents and using web search to extend the available information when the original corpus produces weak results.
The grader does not need to be an unrestricted agent. It can use a structured output such as sufficient, partial, irrelevant or stale, followed by a short explanation. The workflow can then route each result through predefined edges.
Normalise Internal and Web Evidence
A vector database and a Web Search API return different types of data. If those formats are passed directly to the model, it becomes difficult to compare sources, remove duplicates or produce reliable citations.
Both sources should be converted into a shared evidence structure:
{
"source_type": "internal | web",
"title": "Source title",
"url_or_document_id": "Source identifier",
"published_at": "Publication or update date",
"content": "Relevant source passage",
"relevance_score": 0.86,
"supported_claims": [
"Claim supported by this source"
]
}
The exact fields can change, but every evidence item needs a stable identity and a traceable origin.
Web results also require additional processing. Several pages may repeat the same press release, quote the same research paper, or reproduce an announcement without adding independent evidence. Deduplication should therefore consider the underlying source, not only whether the URLs are different.
Search-result snippets may be enough to decide which pages deserve inspection, but they are usually too limited to support important claims. When a claim matters, the pipeline should retrieve the page and preserve the relevant passage.
For a deeper explanation of web evidence and source handling, see Agentic Search: How AI Agents Search, Evaluate, and Cite the Web.
Define Search and Stopping Rules
Once an agent can search repeatedly, it needs rules for when another search is justified.
A follow-up search may be appropriate when a key subquestion has no supporting evidence, two credible sources disagree, the retrieved pages are outdated, or the current results introduce a new term that requires investigation.
The pipeline should also know when to stop. It may finish when every important claim has at least one suitable source, the requested topics have been covered, and another query is unlikely to change the conclusion.
Hard limits remain necessary. A production system should cap the number of searches, inspected pages, tokens, elapsed time or API spend. These limits protect the application when the model keeps reformulating queries without finding anything useful.
NVIDIA’s Agentic RAG Blueprint uses planning, task execution, synthesis and optional verification, while also acknowledging that the agentic path requires additional model calls and latency. That trade-off should be explicit in any implementation.
How to Evaluate the Pipeline
The final answer is only one part of an agentic RAG evaluation. The path used to produce it also matters.
Retrieval relevance measures whether the internal search found the correct documents. Fallback precision measures whether web search was called only when it added value. If the system searches the web for every question, the router is not doing useful work.
Groundedness examines whether the answer is supported by the collected evidence, while citation correctness checks whether each cited source supports the particular claim attached to it. Source quality should account for authority, freshness and independence.
Operational metrics are equally important. Web search, page extraction and repeated tool calls add latency and cost. Agent observability should therefore capture queries, routing decisions, retrieved sources, grader outputs, retries and stopping reasons.
An evaluation set should contain simple internal questions, current questions that require the web, questions that need both sources, and questions for which no reliable answer exists. The last category tests whether the system can stop and acknowledge uncertainty instead of continuing to search indefinitely.
When a Basic RAG Pipeline Is Still Better
Agentic RAG should not be the default architecture for every retrieval task.
If the knowledge base is stable, the questions are predictable and one retrieval step usually finds the necessary context, a basic RAG pipeline will be faster, cheaper and easier to evaluate.
Agentic retrieval becomes valuable when the system must choose among sources, handle multi-step questions, recover from weak retrieval or access current information. The additional complexity should solve an identifiable retrieval problem rather than merely make the architecture appear more advanced.
Conclusion
A Web Search API gives a RAG agent access to current information. The agentic pipeline decides when that information is needed, how it should be combined with internal knowledge, which sources deserve to be trusted and when the research should end.
The most reliable systems do not begin with autonomous tools. They begin with a clear evidence model, controlled routing and measurable stopping conditions.
When those foundations are in place, real-time web search becomes more than a fallback. It becomes a traceable evidence layer for answers that an internal knowledge base could not produce alone.
