How to evaluate a RAG system

• Lesson
• Intermediate
• 40 min read
• Checked 17 Aug 2026

In one sentence

A RAG evaluation asks four questions: Did we find the evidence? Did we use it correctly? Did the user get the right result? Can the system keep doing that safely in production?

A polished demo is not proof that a RAG system is ready for production.

The demo shows that one question worked once.

An evaluation shows what works, what fails, and whether a new version is safer to release.

The four-part mental model

Use this order whenever you test or debug RAG:

1. Retrieval: Did we get the right evidence?
2. Generation: Did the model use that evidence correctly?
3. End-to-end: Did the user get the right result?
4. Production: Is the system fast, affordable, reliable, fresh, and secure?

Part	What goes in	What you inspect	Simple question
Retrieval	User question and searchable corpus	Retrieved and reranked chunks	Did we get the evidence?
Generation	Question, final context, and instructions	Answer and citations	Did we use the evidence correctly?
End-to-end	Complete request	Final user outcome	Did the task succeed?
Production	Live traffic and system state	Traces, latency, cost, errors, freshness, and access decisions	Can we trust it at scale?

Do not compress these into one “RAG score.”

One score hides the location of the failure.

1. Retrieval evaluation

Retrieval evaluation studies the evidence going into the model.

Quick metric map

Metric	Plain question	Best used when
Context relevance	Is this chunk about the user’s question?	You want to grade each retrieved chunk
Context precision	Did useful context appear before noise?	Your framework uses judged context relevance and rank
Context recall	Did the retrieved context cover the claims needed for the reference answer?	You have a trustworthy reference answer or source facts
Precision@k	How many of the first k results are relevant?	You have binary relevance labels
Recall@k	How much of all known relevant evidence appeared in the first k results?	You know the full relevant set
Hit@k	Did at least one relevant result appear?	One useful hit may be enough
MRR	How early did the first relevant result appear?	The first useful result matters most
NDCG@k	Did the most useful results appear near the top?	Relevance has levels such as perfect, useful, weak, and irrelevant
MAP@k	Were relevant results ranked high across many questions?	Several relevant results can exist for each question
Duplicate rate and fact coverage	Did we retrieve varied evidence or repeat the same fact?	The answer needs several different facts

The metric name is not enough.

Always record k, the denominator, the relevance rule, the dataset, and the evaluator version.

One running example

The employee asks:

What is the company’s work-from-home policy?

The reviewed policy contains three answer-bearing facts:

1. Employees may work from home for at most two days each week.
2. A manager must approve the arrangement.
3. Employees on probation are not eligible.

The retriever returns these four chunks:

Rank	Retrieved chunk	Relevant?	Fact covered
1	Employees may work from home for up to two days per week.	Yes	Weekly limit
2	The cafeteria serves lunch from noon to 2 p.m.	No	None
3	Work from home requires manager approval.	Yes	Approval
4	Reserved parking is available to directors.	No	None

This small example will stay with us through the chapter.

Context relevance

Context relevance asks whether a retrieved chunk helps answer the question.

In the example:

• ranks 1 and 3 are relevant;
• ranks 2 and 4 are not relevant.

This can be a human label, a deterministic rule, or a model-judge label.

The evaluator needs a rubric.

“Mentions the same topic” is weaker than “contains evidence needed to answer.”

Precision@k

Precision@k asks how clean the first k results are.

Precision@k = relevant results in the first k / k

For the WFH example:

Precision@4 = 2 relevant results / 4 positions = 0.50

The system found useful evidence.

It also spent half of its context budget on noise.

This book calculates Precision@k only when at least k results were returned. If fewer results come back, record the incomplete retrieval or evaluate with a valid k. Do not silently change the denominator.¹

Recall@k

Recall@k asks how much of the known relevant evidence we found.

Recall@k = relevant results in the first k / all known relevant results

For the WFH example:

Recall@4 = 2 retrieved policy facts / 3 known policy facts = 0.67

The probation rule is missing.

This is the most important fact in the example because it may change the answer for a new employee.

Recall@k needs a trustworthy gold set.

If nobody has identified all relevant evidence, the denominator is unknown.

Hit@k

Hit@k asks a smaller question:

Hit@k = 1 if any relevant result appears in the first k; otherwise 0

For the WFH example:

Hit@4 = 1

That sounds good, but it hides the missing probation rule.

Hit@k is useful when one relevant passage is sufficient.

It is too weak when an answer needs several facts.

MRR

Reciprocal rank asks where the first relevant result appeared.

Reciprocal rank = 1 / rank of the first relevant result
MRR = mean reciprocal rank across all test questions

The first result in the WFH example is relevant.

Its reciprocal rank is 1 / 1 = 1.0.

If the first relevant result appeared at rank 3, the reciprocal rank would be 1 / 3 = 0.33.

MRR ignores every relevant result after the first.

Pair it with Recall@k when the answer needs multiple passages.

NDCG@k

Binary labels say only “relevant” or “not relevant.”

Real search results are often more nuanced.

A chunk may answer the question completely, support one condition, mention the topic weakly, or be useless.

NDCG@k supports graded relevance.

It gives more credit when highly useful results appear near the top.

It compares the returned order with an ideal order in which the strongest evidence comes first.²

Use NDCG when rank and degree of usefulness both matter.

MAP@k

Average Precision@k looks at every rank where a relevant result appears for one question.

It rewards a list that places relevant results early and consistently.

Mean Average Precision@k, or MAP@k, averages that value across the test questions.¹

Use MAP when each question can have several relevant results.

Use MRR when the first relevant result is the main goal.

Context precision and context recall

Evaluation libraries may use these names differently from textbook Precision@k and Recall@k.

For example, Ragas context precision examines whether relevant retrieved chunks are ranked above irrelevant ones. Ragas context recall examines how many claims in a reference answer are supported by retrieved context.³

That means the denominator may be:

• returned positions for Precision@k;
• known relevant passages for Recall@k;
• judged chunk ranks for a context-precision evaluator;
• reference-answer claims for a context-recall evaluator.

Record the library, metric version, inputs, and rubric beside the score.

Redundancy and diversity

Imagine the retriever returns four chunks that all repeat the two-day limit.

Precision may look high because all four chunks are relevant.

The answer can still miss manager approval and the probation rule.

Track at least two additional values:

Check	Simple calculation	What it reveals
Duplicate rate	near-duplicate results / returned results	Wasted context space
Unique fact coverage	distinct gold facts found / all gold facts	Whether the evidence set covers different needs

MMR can select results that balance relevance and novelty.

MMR is a retrieval or reranking method, not an evaluation metric.

It cannot recover a relevant chunk that never entered the candidate set.

Runnable retrieval example

The example below keeps the denominators visible in code.

src/examples/evals/retrieval_metrics.py

"""Transparent retrieval metrics with explicit denominators."""


def validate_k(retrieved: list[str], k: int) -> None:
    if k <= 0:
        raise ValueError("k must be positive")
    if len(retrieved) < k:
        raise ValueError("retrieved must contain at least k results")


def precision_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    validate_k(retrieved, k)
    top_k = retrieved[:k]
    relevant_retrieved = sum(document_id in relevant for document_id in top_k)
    return relevant_retrieved / k


def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    validate_k(retrieved, k)
    if not relevant:
        return 0.0
    relevant_retrieved = sum(document_id in relevant for document_id in retrieved[:k])
    return relevant_retrieved / len(relevant)


retrieved_ids = ["refund-policy", "pricing", "security"]
relevant_ids = {"refund-policy", "cancellation-policy"}

print(precision_at_k(retrieved_ids, relevant_ids, k=3))  # 1 / 3
print(recall_at_k(retrieved_ids, relevant_ids, k=3))     # 1 / 2

2. Generation evaluation

Generation evaluation studies the answer coming out of the model.

Quick metric map

Metric	Plain question	Compare the answer with
Faithfulness or groundedness	Does the evidence support every claim?	Retrieved context
Answer relevance	Did the model answer what the user asked?	User question
Correctness	Does the answer agree with trusted truth?	Reference answer or reviewed facts
Completeness	Did the answer include every decision-changing fact?	Required facts or reference answer
Citation correctness	Does each citation support its attached claim?	Cited passage
Citation coverage	Are important claims cited?	Claims in the answer
Conciseness	Is the answer direct without losing meaning?	Style rubric and task need
Coherence	Is the answer logically organized and readable?	Structure rubric
Instruction following	Did it follow the requested format and constraints?	User and system instructions
Safety	Did it avoid prohibited or unauthorized behavior?	Deterministic policy and reviewed cases
Abstention	Did it refuse to guess when evidence was insufficient?	Answerability label and evidence threshold

LangSmith’s RAG guide treats correctness, relevance, groundedness, and retrieval relevance as separate comparisons.⁴

That separation is useful because one answer can pass one check and fail another.

Continue the WFH example

Assume the final context now contains all three policy facts.

The model answers:

Employees can work from home up to two days per week with manager approval.

Now score the answer:

Check	Result	Reason
Faithfulness	Pass	Both claims appear in the context
Answer relevance	Pass	It directly answers the WFH question
Correctness	Partial or fail	The statement is true but omits an eligibility restriction
Completeness	Fail	It leaves out the probation rule
Citation correctness	Depends on citation	The cited passage must support both included claims
Citation coverage	Depends on product rule	Important policy claims should be traceable

This is why “faithful” does not mean “complete.”

The model did not invent a fact.

It still left out a fact that could change the employee’s decision.

A better answer is:

Employees may work from home up to two days per week with manager approval. Employees on probation are not eligible. [Policy §4.2]

Four answers, four diagnoses

Use this smaller return-policy case to see how the labels separate:

Question: What is the return window?

Retrieved context:
- Returns are accepted for 30 days.
- The company was founded in 1998.

Trusted reference:
The return window is 30 days, and a receipt is required.

Generated answer	Faithful?	Relevant?	Correct?	Diagnosis
“Returns are accepted for 30 days.”	Yes	Yes	Yes	Direct and supported, but check completeness separately
“Returns are accepted for 30 days without a receipt.”	No	Yes	No	The receipt claim is unsupported and contradicts the reference
“The company was founded in 1998.”	Yes	No	No	Supported by context, but unrelated to the question
“Returns are accepted for 60 days.”	No	Yes	No	It answers the topic but contradicts evidence and truth

A grounded answer can still be wrong when the indexed source is stale.

A correct answer can be ungrounded when the model knows a fact that the supplied evidence did not contain.

Keep both checks.

LLM-as-a-judge setup

A model judge can help score qualities that are hard to express as code.

Give the judge only the inputs needed for the metric:

1. Question — what the user asked.
2. Retrieved context — the evidence available to the answer model.
3. Generated answer — the output being evaluated.
4. Reference answer or required facts — include these only when the metric needs trusted truth.
5. Rubric — define what pass, partial, and fail mean.

Do not ask one vague question such as “Is this a good answer?”

Ask one narrow question per score.

Score	Minimum judge inputs	Example judge question
Faithfulness	Context and answer	Does the context support every answer claim?
Relevance	Question and answer	Does the answer directly address the question?
Correctness	Question, answer, and reference	Does the answer agree with the reviewed facts?
Completeness	Answer and required facts	Which required facts are missing?
Citation correctness	Answer and cited passages	Does each cited passage support its attached claim?

Use deterministic code for exact checks such as JSON shape, required keys, allowed citation IDs, and length limits.

Calibrate model judges against a human-reviewed set.

Version the judge model and rubric.

Inspect disagreement cases.

A judge is another fallible model, not ground truth.⁵

src/examples/evals/rag_answer_metrics.py

"""Keep retrieval scores separate from answer-quality judgments."""

from dataclasses import dataclass
from typing import Protocol


@dataclass(frozen=True)
class EvalCase:
    question: str
    relevant_passage_ids: set[str]
    reference_answer: str


@dataclass(frozen=True)
class RagRun:
    retrieved_passage_ids: list[str]
    context: str
    answer: str


@dataclass(frozen=True)
class AnswerGrades:
    grounded: bool
    relevant: bool
    correct: bool


class AnswerGrader(Protocol):
    def grade(self, case: EvalCase, run: RagRun) -> AnswerGrades: ...


def evaluate_rag_run(
    case: EvalCase, run: RagRun, *, k: int, grader: AnswerGrader
) -> dict[str, float | bool | None]:
    if k <= 0 or len(run.retrieved_passage_ids) < k:
        raise ValueError("k must be positive and no larger than the result list")

    top_k = run.retrieved_passage_ids[:k]
    relevant_found = sum(item in case.relevant_passage_ids for item in top_k)
    grades = grader.grade(case, run)
    recall = (
        relevant_found / len(case.relevant_passage_ids)
        if case.relevant_passage_ids
        else None
    )

    return {
        "precision_at_k": relevant_found / k,
        # Recall is undefined when the case has no labeled relevant passages.
        "recall_at_k": recall,
        "grounded": grades.grounded,
        "answer_relevant": grades.relevant,
        "correct": grades.correct,
    }

Evaluate a generated summary

A summary can sound natural while dropping a condition.

Source:
Hotel accommodation is provided only when an overnight stay is required
and the disruption was caused by the airline.

Bad summary:
Passengers receive a hotel after a long delay.

The summary is short.

It is also wrong.

It replaced two required conditions with the vague phrase “long delay.”

Check summaries in this order:

1. Claim support: Does the source support every claim?
2. Required-fact coverage: Did every decision-changing condition survive?
3. Contradiction: Did the summary reverse or weaken a rule?
4. Citation correctness: Does each citation support its claim?
5. Citation coverage: Can a reviewer trace the important claims?
6. Concision: Did the summary remove repetition without removing meaning?
7. Uncertainty: Did it say when the evidence was incomplete?

Reference-overlap metrics can be useful supporting signals.

Similar wording does not prove factuality.

Different wording does not prove an error.

3. End-to-end and production evaluation

Component scores tell you where the system is weak.

End-to-end evaluation tells you whether the complete product helped the user.

Follow one airline request

This airline rule is fictional and exists only for the lesson.

The passenger asks:

My flight is delayed by six hours. Am I eligible for a free hotel?

The policy says a hotel is provided only when both conditions are true:

1. The delay requires an overnight stay.
2. The airline caused the delay.

The user has not supplied either fact.

The safe answer explains the two conditions and asks for the missing information.

Run	Retrieval result	Generated answer	End-to-end result
Retrieval failure	Correct policy absent	Any answer lacks the required evidence	Fix retrieval first
Generation failure	Correct policy present	“Yes. Six hours qualifies.”	Block: relevant but unfaithful and incorrect
Passing run	Correct policy present	Explains both conditions and asks for missing facts	Allow: useful and appropriately uncertain

src/examples/evals/flight_policy_eval.py

"""Separate retrieval failure from generation failure in a tiny RAG eval."""

from dataclasses import dataclass


QUESTION = "My flight is delayed by six hours. Am I eligible for a free hotel?"
GOLD_POLICY = (
    "Hotel accommodation is provided only when the delay requires an overnight "
    "stay and the airline caused the delay."
)
RELEVANT_POLICY_IDS = {"hotel-policy"}
K = 3


@dataclass(frozen=True)
class AnswerLabels:
    # None means the isolated generation check was intentionally not run.
    faithfulness: bool | None
    answer_relevance: bool | None
    correctness: bool | None


@dataclass(frozen=True)
class EvalRun:
    name: str
    retrieved_ids: list[str]
    answer: str
    labels: AnswerLabels


@dataclass(frozen=True)
class RetrievalScores:
    relevant_found: int
    precision_denominator: int
    recall_denominator: int
    hit_at_k: int
    precision_at_k: float
    recall_at_k: float


def retrieval_metrics(
    retrieved_ids: list[str], relevant_ids: set[str], k: int
) -> RetrievalScores:
    """Return both fractions and scores so the denominators stay visible."""
    if k <= 0 or len(retrieved_ids) < k:
        raise ValueError("k must be positive and no larger than the result list")
    if not relevant_ids:
        raise ValueError("use an abstention eval when no relevant item exists")

    top_k = retrieved_ids[:k]
    if len(set(top_k)) != k:
        raise ValueError("the top-k list must not contain duplicate IDs")

    relevant_found = sum(item in relevant_ids for item in top_k)
    return RetrievalScores(
        relevant_found=relevant_found,
        precision_denominator=k,
        recall_denominator=len(relevant_ids),
        hit_at_k=int(relevant_found > 0),
        precision_at_k=relevant_found / k,
        recall_at_k=relevant_found / len(relevant_ids),
    )


def diagnose(run: EvalRun) -> str:
    scores = retrieval_metrics(run.retrieved_ids, RELEVANT_POLICY_IDS, K)

    print(f"\n{run.name}")
    print(f"  Hit@{K}:       {scores.hit_at_k}")
    print(
        f"  Precision@{K}: {scores.relevant_found}/"
        f"{scores.precision_denominator} = {scores.precision_at_k:.3f}"
    )
    print(
        f"  Recall@{K}:    {scores.relevant_found}/"
        f"{scores.recall_denominator} = {scores.recall_at_k:.3f}"
    )
    print(f"  Answer: {run.answer}")
    print(f"  Answer labels: {run.labels}")

    if scores.recall_at_k < 1.0:
        return "retrieval failure: the gold policy never reached the model"
    if run.labels != AnswerLabels(True, True, True):
        return "generation failure: retrieval passed, but an answer label failed"
    return "case-level pass: needed evidence and all answer labels passed"


runs = [
    EvalRun(
        name="Retrieval failure",
        retrieved_ids=["meal-policy", "baggage-policy", "refund-policy"],
        answer="Generation is not scored in this isolated retrieval test.",
        labels=AnswerLabels(None, None, None),
    ),
    EvalRun(
        name="Generation failure",
        retrieved_ids=["hotel-policy", "meal-policy", "baggage-policy"],
        answer="Yes. A six-hour delay qualifies for a free hotel.",
        labels=AnswerLabels(
            faithfulness=False,
            answer_relevance=True,
            correctness=False,
        ),
    ),
    EvalRun(
        name="Passing run",
        retrieved_ids=["hotel-policy", "meal-policy", "baggage-policy"],
        answer=(
            "A six-hour delay alone does not establish eligibility. Does the delay "
            "require an overnight stay, and has the airline confirmed that it caused "
            "the delay?"
        ),
        labels=AnswerLabels(
            faithfulness=True,
            answer_relevance=True,
            correctness=True,
        ),
    ),
]


if __name__ == "__main__":
    print(f"Question: {QUESTION}")
    print(f"Gold policy: {GOLD_POLICY}")
    for eval_run in runs:
        print(f"  Diagnosis: {diagnose(eval_run)}")

Production scorecard

Offline evaluation is evidence for readiness.

It is not proof of readiness by itself.

Area	What to measure	Example failure
Task outcome	Task success and human review	Answer is technically true but not useful
Latency	p50, p95, and timeout rate by stage	Reranking pushes p95 beyond the SLA
Cost	Cost per query and cost per successful answer	A more accurate version costs too much to operate
Reliability	Error, fallback, empty-retrieval, and retry rates	Reranker timeouts silently return weak ordering
Freshness	Source age, indexing lag, and stale-answer rate	Deleted policy remains searchable
Security	ACL leakage tests and unauthorized passage exposure	Retrieval returns another tenant’s chunk
Citations	Supported-claim rate and broken-source rate	Citation exists but does not support the sentence
Drift	Score distributions and failure clusters over time	A new query pattern lowers recall

Cost per successful answer is usually more meaningful than cost per request.

A cheap wrong answer is not a saving.

4. Debug low retrieval quality in order

When a user reports a bad answer, do not begin by changing the prompt.

Trace the evidence path.

Step 1: Confirm the failing stage

Inspect the final context that reached the model.

• If the needed evidence is absent, investigate ingestion, retrieval, reranking, or context packing.
• If the evidence is present but the answer misuses it, investigate generation.

Start with this question:

Did we retrieve bad evidence, or did we use good evidence badly?

Step 2: Confirm that the source exists

Ask whether the corpus contains the answer at all.

Check the correct document version, effective date, tenant, language, and permission scope.

Retrieval cannot return a policy that was never ingested or is excluded by access rules.

Step 3: Inspect parsing, chunks, and metadata

Open the stored chunk, not only the original document.

Check:

• whether PDF text and tables were parsed correctly;
• whether a chunk boundary split the answer;
• whether headers and parent sections were preserved;
• whether metadata and ACL fields are correct;
• whether updates and deletions reached every index.

No chunking strategy is always best.

Fixed-size, recursive, semantic, and structure-aware chunking must earn their place on the same labeled evaluation set.

Step 4: Inspect the original and rewritten query

Log both versions.

A rewrite may improve a vague question.

It may also remove an exact error code, product ID, date, negation, or user constraint.

Keep the original query available for comparison and hybrid retrieval.

Step 5: Verify the embedding contract

Query and document vectors must come from the same compatible embedding space.

Check the model name, version, dimensions, preprocessing, and distance metric.

Inspect score distributions as a diagnostic signal.

Do not treat close cosine scores as proof of “embedding collapse.”

Similar scores can also come from a hard query, a homogeneous corpus, normalization, or the index configuration.

Step 6: Check filters and index behavior

Run the same query with and without safe diagnostic filters.

Check whether tenant, date, product, or permission filters removed the gold passage.

For an approximate vector index, compare its top results with exact search on a labeled sample.

This measures ANN recall against exact neighbors.

It is different from human-labeled RAG Recall@k.

Step 7: Label the top results

Print the top 10 or 20 candidates with IDs, scores, text, and metadata.

Label them as strongly relevant, partly relevant, or irrelevant.

Then compute the metric that matches the need:

• Precision@k for noise;
• Recall@k for missing evidence;
• MRR for the first useful result;
• NDCG for graded ranking;
• duplicate rate and unique fact coverage for context variety.

Step 8: Compare retrieval, fusion, and reranking

Save the list at every boundary:

1. keyword candidates;
2. vector candidates;
3. fused candidates;
4. reranked candidates;
5. final packed context.

Compare before and after each step.

A reranker can improve ordering.

It can also remove a useful result, time out, or fall back to the raw ranking.

Step 9: Inspect context packing

The retriever may find the right passage and still lose it later.

Check token limits, deduplication, parent-child expansion, ordering, and truncation.

The final context sent to the model is the real generation input.

Step 10: Trace and monitor the complete path

For each failing query, preserve:

1. User question
2. Original and rewritten query
3. Corpus, embedding, and index version
4. Retrieved candidates
5. Fused and reranked results
6. Final packed context
7. Generated answer and citations
8. Metric labels and evaluator version
9. Latency, cost, errors, and fallback decisions

Monitor empty retrievals, labeled Hit@k and Recall@k samples, latency, index freshness, fallback rates, authorization failures, and recurring query clusters.

Do not call an unlabeled live score “precision” unless its relevance labels and denominator are clear.

Debugging order to remember

Source → Parse → Chunk → Query → Filter → Retrieve → Rerank → Pack → Answer

There is no honest universal claim that most failures come from one stage.

Measure your system and fix the stage your trace identifies.

5. Build a trustworthy evaluation

Build the dataset before the dashboard

A useful first dataset can be small.

1. Collect real questions from users and domain experts.
2. Include ordinary, multi-passage, exact-identifier, unanswerable, stale-data, and adversarial cases.
3. Label acceptable evidence, required facts, prohibited conclusions, and answer behavior.
4. Include different tenants, languages, permission levels, and source types when the product supports them.
5. Preserve dataset versions.
6. Inspect individual failures before averaging scores.

Use human-reviewed examples to calibrate automated judges.

Compare versions with the same test

Freeze the questions, corpus and permission snapshot, k, query transforms, generation settings, reference facts, evaluator version, and thresholds.

If chunking changes, freeze gold source spans or answer-bearing facts.

Map each candidate’s chunks back to those stable units.

Chunk IDs are not stable labels when chunk boundaries change.

Check	Baseline	Candidate	Decision rule
Recall@5	measured value	measured value	Candidate must meet the retrieval gate
Precision@5	measured value	measured value	Candidate must not fill context with noise
Faithful-answer rate	measured value	measured value	Candidate must meet the grounding gate
Correct-answer rate	measured value	measured value	Review every high-impact regression
p95 latency	measured value	measured value	Candidate must stay inside the SLA
Cost per successful answer	measured value	measured value	Candidate must stay inside the budget

If chunking, embeddings, and reranking all change together, the test supports a claim about the whole candidate bundle.

Use one-factor changes or ablations before crediting one component.

Do not publish invented improvements such as “recall rose from 70% to 90%.”

Report measured values, dataset size, percentage-point change, important regressions, and uncertainty.

Turn metrics into release gates

Set the decision rule before running the test.

These values are examples, not universal targets:

Release only if:
  # Retrieval
  retrieval recall@5 >= team threshold
  retrieval precision@5 >= team threshold

  # Answer behavior
  faithful answer rate >= team threshold
  reference correctness >= team threshold
  correct abstention rate >= team threshold
  supported citation rate >= team threshold

  # Safety
  unauthorized passage exposure == 0
  critical prompt-injection successes == 0

  # Operations
  p95 latency <= agreed SLA
  cost per successful answer <= agreed budget
  request error rate <= agreed SLO

An average quality score cannot excuse a tenant-data leak.

Safety failures are hard stops.

Interview answer

I evaluate RAG in four parts. First, retrieval: did the system find the required evidence, and did it rank that evidence well? I use metrics such as Precision@k, Recall@k, Hit@k, MRR, and NDCG on a labeled dataset. Second, generation: did the model answer the question faithfully, correctly, completely, and with supporting citations? Third, end-to-end behavior: did the user get the right outcome, including a safe abstention when evidence was missing? Finally, production: I monitor p95 latency, cost per successful answer, failures, freshness, and authorization. I keep these scores separate so a bad answer tells me whether to fix ingestion, retrieval, context packing, generation, or operations.

The shortest version is:

• RetrievalDid we get the right evidence?
• GenerationDid we use the evidence correctly?
• End-to-endDid the user get the right result?
• ProductionCan we keep doing it safely and reliably?

FDE review questions

• What user decision could a wrong answer change?
• Which facts are required before the system may answer?
• Which failures are severe even if the average score is high?
• Can a reviewer trace a failing score to a source, query, chunk, model output, and code version?
• Does every metric name include its denominator, labels, dataset, and evaluator?

Footnotes

1. Google for Developers defines Precision@k, Average Precision@k, and MAP@k for ranked lists. Microsoft documents retrieval-specific Recall@k and MRR in its RAG information-retrieval guidance. ↩ ↩²

2. Microsoft Foundry describes MAP and NDCG as ranking metrics and explains that NDCG compares returned rankings with an ideal relevance order. ↩

3. Ragas documents its definitions of context precision and context recall. Check the exact variant and inputs used by your installed version. ↩

4. LangChain’s RAG evaluation tutorial separates answer correctness, answer relevance, groundedness, and retrieval relevance. ↩

5. LangChain’s evaluation concepts distinguish reference-based and reference-free evaluators, along with human, code, and model-based evaluation. ↩