Agentic RAG earns its loop

• Lesson
• Advanced
• 18 min read
• Checked 15 Aug 2026

In one sentence

Agentic RAG lets model decisions change retrieval while a request is running. Use that flexibility only when evidence shows a fixed or routed workflow cannot handle the query.

Suppose an enterprise asks:

Compare three model families for our RAG use case using internal requirements, current benchmark results, and our infrastructure cost data.

The answer needs several kinds of evidence. That does not prove it needs several agents.

Three retrieval shapes

Shape	Who selects the sources and next step?	Good fit
Fixed RAG	Application code runs a predefined retriever	One known corpus and one-hop questions
Routed or parallel RAG workflow	Code or a bounded classifier selects from known branches	Sources are known and can be searched independently
Agentic or adaptive RAG	A model changes queries, sources, or stopping decisions from observations	Open-ended, multi-hop research whose path is unknown in advance

“Traditional RAG searches one vector database” is too narrow. The original RAG pattern combines retrieval with generation, not one mandatory store.¹ A fixed pipeline can query SQL, keyword search, APIs, and a vector index.

Agentic RAG is also not one standard architecture. It is a useful name for systems where model-driven decisions influence whether, where, or how often to retrieve. Research systems such as Adaptive-RAG choose among no retrieval, one retrieval step, and iterative retrieval according to estimated question complexity.²

Start with the known workflow

For the model-comparison question, the required evidence is already clear:

1. 01Decompose required evidence
2. 02Authorize sources
3. 03Retrieve independent facts in parallel
4. 04Validate coverage and freshness
5. 05Normalize trade-offs
6. 06Synthesize with citations

internal requirements ─┐
current eval results ──┼─ validate provenance and comparable metrics ─ recommendation
internal cost data ────┘

This fan-out can be deterministic. It is cheaper to test than asking a planner model to rediscover the same three subtasks on every request.

Use model-directed retrieval when the evidence path genuinely depends on findings. For example, a benchmark result may mention a failure mode that requires a new targeted search, or two sources may use incompatible task definitions that require clarification.

Roles are not necessarily agents

Label often used in diagrams	It can be implemented as
Planner	A function, template, classifier, or model call that proposes subtasks
Router	An allowlisted mapping from subtask type to source
Retriever	A normal search client or API call
Reflection or coverage check	Required-field validation, a rubric, or a model-assisted gap check
Synthesizer	One grounded generation call over validated evidence

Do not create five networked agents merely because a diagram has five boxes. Anthropic distinguishes fixed workflows from agents and recommends adding complexity only when task performance justifies its latency and cost.³

Evidence must survive orchestration

Every retrieval worker should return a typed evidence record:

{
  "subquestion": "What does our internal RAG eval show?",
  "source": "benchmark://rag-golden-set/run-184",
  "observed_at": "2026-08-15",
  "statement": "Model B passed 94 of 100 cases",
  "metric_definition": "project-specific accepted answer rubric"
}

Keep the source, access scope, date, dataset version, metric definition, and raw result ID. A model’s relevance score is not provenance.

If one route uses web search, allowlist appropriate primary sources, keep the exact page URL plus publication and retrieval dates, and treat vendor claims separately from independent or internal measurements. Never place private requirements or cost rows into a public search query. Public benchmarks help discover candidates; the company’s own representative evaluation set should drive the product decision.

The final comparison should put constraints on the same scale:

• quality on the company’s own representative cases;
• p50 and p95 latency at the expected concurrency;
• cost per accepted grounded answer, not only token price;
• data handling, region, retention, and security requirements;
• tool support, rate limits, and operational fit;
• source date and model version.

A public benchmark saying one model is “best” and internal cost data saying it is expensive are not contradictory agent decisions. They are different measurements in a multi-objective decision.

Build the deterministic baseline first

The following standard-library example uses synthetic evidence. It runs three known retrieval tasks in parallel, allowlists the sources, and refuses to continue if required evidence is missing, stale, or lacks provenance.

src/examples/rag/multi_source_research_workflow.py

"""A deterministic multi-source research workflow.

All records below are synthetic fixtures. The example shows orchestration and
evidence checks; it does not compare real model quality or prices.
"""

from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass
from datetime import date


@dataclass(frozen=True)
class ResearchTask:
    question: str
    source: str


@dataclass(frozen=True)
class Evidence:
    source: str
    locator: str
    observed_at: str
    statement: str


ALLOWED_SOURCES = frozenset({"internal_docs", "benchmark_registry", "cost_sql"})
MAX_AGE_DAYS = {
    "internal_docs": 90,
    "benchmark_registry": 7,
    "cost_sql": 45,
}

FIXTURES = {
    "internal_docs": Evidence(
        source="internal_docs",
        locator="policy://ai/model-requirements-v3#security",
        observed_at="2026-08-15",
        statement="The selected path must support regional processing and citations.",
    ),
    "benchmark_registry": Evidence(
        source="benchmark_registry",
        locator="benchmark://rag-golden-set/run-184",
        observed_at="2026-08-15",
        statement="Model B passed 94 of 100 synthetic evaluation cases.",
    ),
    "cost_sql": Evidence(
        source="cost_sql",
        locator="sql://finance/model-costs/2026-07",
        observed_at="2026-08-01",
        statement="Model B cost 18 synthetic units per accepted answer.",
    ),
}


def make_plan() -> list[ResearchTask]:
    """These known requirements do not need an LLM planner."""
    return [
        ResearchTask("What constraints must the model satisfy?", "internal_docs"),
        ResearchTask("How does each candidate perform on our RAG eval set?", "benchmark_registry"),
        ResearchTask("What is the cost per accepted grounded answer?", "cost_sql"),
    ]


def retrieve(task: ResearchTask) -> Evidence:
    if task.source not in ALLOWED_SOURCES:
        raise PermissionError(f"source is not approved: {task.source}")
    return FIXTURES[task.source]


def validate_evidence(rows: list[Evidence], *, as_of: date) -> None:
    found = {row.source for row in rows}
    missing = ALLOWED_SOURCES - found
    if missing:
        raise RuntimeError(f"human review: missing evidence from {sorted(missing)}")
    if any(not row.locator or not row.observed_at for row in rows):
        raise RuntimeError("human review: evidence lacks provenance or date")
    for row in rows:
        try:
            observed_at = date.fromisoformat(row.observed_at)
        except ValueError as exc:
            raise RuntimeError("human review: evidence has an invalid date") from exc
        age_days = (as_of - observed_at).days
        if age_days < 0 or age_days > MAX_AGE_DAYS[row.source]:
            raise RuntimeError(f"human review: stale evidence from {row.source}")


def research() -> list[Evidence]:
    plan = make_plan()
    with ThreadPoolExecutor(max_workers=len(plan)) as pool:
        rows = list(pool.map(retrieve, plan))
    validate_evidence(rows, as_of=date(2026, 8, 15))
    return rows


if __name__ == "__main__":
    for evidence in research():
        print(f"[{evidence.source}] {evidence.statement}")
        print(f"  source: {evidence.locator} ({evidence.observed_at})")

Run it on Windows from an activated Python 3.10+ virtual environment:

python --version
python src/examples/rag/multi_source_research_workflow.py

On macOS or Linux:

python3 --version
python3 src/examples/rag/multi_source_research_workflow.py

This is deliberately not agentic: application code owns the plan, sources, and stopping rule.

Add one bounded adaptive decision

An agentic extension can let a model propose an extra ResearchTask only after the deterministic coverage check identifies a gap:

for round_number in range(2):
    gaps = find_evidence_gaps(evidence)
    if not gaps:
        break

    proposed_task = model_proposes_one_search(gaps)
    validated_task = validate_source_query_and_budget(proposed_task)
    evidence.append(retrieve(validated_task))

if find_evidence_gaps(evidence):
    send_to_human_review("evidence is still incomplete")

The model changes the next retrieval step from an observation, so this small section is agentic. The controller still validates the source, caps rounds and cost, deduplicates queries, and stops or escalates when coverage remains incomplete.

What can break

Failure	Control
Planner omits the internal security requirement	Deterministic required-evidence checklist
Router sends private data to web search	Source allowlist and data-classification policy
Worker cites a stale benchmark	Freshness rule and versioned source metadata
Reflection says “complete” without cost data	Code checks required evidence types
Several workers repeat the same search	Query budget and deduplication
Synthesis compares unlike metrics	Normalize dataset, rubric, model version, and denominator
Loop keeps researching	Maximum rounds, time, tokens, and explicit stop reasons

ReAct shows why an observation loop can help multi-step tasks: later actions can respond to earlier evidence.⁴ The same loop adds failure paths, so evaluate plan coverage, source choice, evidence quality, citation correctness, task success, latency, and cost separately.

Interview answer

Fixed RAG uses a predefined retrieval path. Agentic RAG allows model decisions to change what is retrieved, from where, or whether another retrieval step is needed. I begin with a fixed multi-source workflow and add that loop only when measured cases require an adaptive evidence path.

Next: production retrieval, agent systems, or how to evaluate RAG.

Footnotes

1. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”. ↩

2. Jeong et al., “Adaptive-RAG”, evaluates routing questions among no-retrieval, single-step, and iterative strategies and also reports classifier errors. ↩

3. Anthropic, “Building effective agents”, describes workflows, routing, parallelization, and model-directed agents as different patterns with cost and control trade-offs. ↩

4. Yao et al., “ReAct: Synergizing reasoning and acting in language models”. ↩