RAG evaluation is the process of testing whether a retrieval-augmented generation system finds the right information, uses it faithfully, and produces useful answers. This guide provides a reusable evaluation framework for building datasets, measuring retrieval quality, reviewing grounded answers, running regression tests, and deciding when an evaluation suite needs to change.
Overview
A RAG application has at least two connected quality problems: retrieval and generation. The retriever must identify relevant passages from a knowledge base, while the language model must use those passages accurately and follow the application’s response requirements. A fluent answer can still be wrong if the retrieved context is incomplete, outdated, or irrelevant.
For that reason, RAG evaluation should not be reduced to a single score. A practical evaluation framework separates the system into observable stages:
- Question quality: Is the test question clear, realistic, and answerable from the available knowledge base?
- Retrieval quality: Did the search return the passages that contain the information needed to answer?
- Grounding: Are the claims in the answer supported by the retrieved context?
- Answer quality: Is the response correct, complete, relevant, and appropriately formatted?
- Operational behavior: Does the application handle no-result queries, conflicting documents, citations, latency, and failures as intended?
This separation makes debugging more practical. If retrieval is weak, prompt optimization alone is unlikely to solve the problem. If retrieval is strong but the response introduces unsupported claims, the generation instructions, context formatting, or output validation may need attention. For broader architecture decisions, see Fine-Tuning vs Prompt Engineering vs RAG.
Template structure
A maintainable RAG evaluation set should record more than a question and an expected answer. Use a structured test case so each result can be explained and reproduced.
{
"case_id": "billing-001",
"question": "How can an administrator change the billing contact?",
"question_type": "procedural",
"expected_answer": "The administrator changes the billing contact from the account billing settings.",
"required_facts": [
"The user must have administrator access",
"The billing settings contain the contact option"
],
"relevant_document_ids": ["billing-guide-v3"],
"allowed_sources": ["product-documentation"],
"expected_behavior": "answer_with_citation",
"risk_level": "medium",
"notes": "Reject answers that suggest changing the contact through profile settings."
}The question represents the user input. Required facts identify the information an acceptable answer must contain, which is more useful than relying on one exact wording. Relevant document IDs support retrieval analysis. Expected behavior covers cases where the correct response is not a direct answer: the system may need to ask for clarification, state that the knowledge base does not contain enough information, or refuse to use an unauthorized source.
Organize cases by scenario rather than creating a random collection of questions. Useful groups include factual lookups, multi-step procedures, comparisons, troubleshooting, ambiguous requests, unanswerable questions, conflicting-document cases, and questions requiring a particular citation or output format. Include both common workflows and failure-prone edge cases.
Metrics to track
For retrieval, compare the returned passages with the passages marked as relevant. Recall at k asks whether the needed evidence appears within the top k results. Precision at k asks how much of the returned context is relevant. These measures help identify whether the search returns too little evidence or too much distracting material. Mean reciprocal rank can be useful when the position of the first relevant result matters.
For generated answers, evaluate faithfulness or groundedness: whether claims are supported by the retrieved context. Also assess answer correctness, completeness, relevance, citation accuracy, and adherence to the required format. A response can be faithful but incomplete, or complete-looking but unsupported. Keep those dimensions separate where possible.
How to customize
Start with a small, representative dataset instead of attempting to model every possible query. Ask subject-matter reviewers to identify the tasks that matter most to users and the errors that would create the greatest operational or compliance risk. Label those cases clearly so a weighted score does not hide serious failures in high-risk scenarios.
Define pass criteria before running the system. For example, a procedural answer may pass only when it includes the correct action, the required permission, and a source reference. A troubleshooting response may pass when it distinguishes confirmed steps from suggestions and does not claim that an unavailable diagnostic was performed.
Use a combination of automated checks and human review. Automated checks can compare retrieved document IDs, test required fields, detect empty answers, and verify that citations point to returned context. Human reviewers are better suited to judging nuanced correctness, whether an explanation is misleading, and whether the response handles ambiguity appropriately.
When using model-based evaluators, provide a clear rubric, the question, the retrieved context, and the answer being assessed. Ask for a structured result with a score, pass or fail decision, and concise reason. Treat that result as an evaluation signal rather than unquestionable truth. Periodically compare evaluator decisions with human judgments, especially for high-impact cases.
Keep retrieval and generation experiments separate. Test chunk size, metadata filters, query rewriting, reranking, and embedding changes against the same dataset. Then test prompt changes, model changes, context ordering, and output constraints. This makes it easier to determine what caused an improvement or regression. For related implementation practices, the prompt versioning guide can help connect evaluation results to specific prompt revisions.
Examples
Example: a strong retrieval result with a weak answer
Suppose the top results contain the correct account-recovery procedure, including an administrator requirement. The generated response says, “Any user can reset the account from the profile menu.” Retrieval recall is good, but the answer is not grounded in the context and fails the required-facts check. The likely investigation targets are context presentation, instructions, output validation, and model behavior—not the search index alone.
Example: a fluent answer with missing evidence
A user asks whether a feature supports a particular file type. The retriever returns general product documentation but no passage that addresses file support. The model gives a confident yes. This case should fail groundedness and answerability checks. The desired behavior may be to state that the available documentation does not confirm the feature, then direct the user to an approved source or request more context.
Example: conflicting documents
Two documents describe different retention periods because one is older. The evaluation case should identify the preferred source or document version and require the answer to acknowledge the conflict when it cannot be resolved. This tests metadata filtering, source priority, version handling, and citation behavior together.
For applications that process transcripts or spoken requests, include errors introduced before retrieval. A voice note may be transcribed incorrectly, changing the search intent. The AI transcription comparison provides relevant background for evaluating that upstream part of the workflow. Similarly, text similarity methods can help compare retrieved passages or detect duplicate knowledge-base content; see text similarity APIs and libraries.
When to update
Revisit the evaluation suite whenever the inputs or behavior of the application change. Important triggers include a new embedding model, chunking strategy, reranker, language model, system prompt, citation format, knowledge source, access-control rule, or document-ingestion pipeline. Changes in user questions, product terminology, and support workflows also justify new cases.
Maintain a stable core set for comparison over time, then add a growing set of recently observed failures. Keep failed cases permanently unless the underlying requirement is intentionally retired. Record the application version, knowledge-base snapshot, retrieval configuration, prompt version, model identifier, and evaluation timestamp with each run. Without this context, a score change is difficult to interpret.
A practical workflow is to run a quick smoke set during development, the full regression set before release, and a scheduled review of production feedback. Investigate large score changes by metric rather than relying on an overall average. If retrieval recall falls, inspect indexing and search changes. If groundedness falls while retrieval remains stable, inspect generation and context handling. If answer quality falls only for new terminology, update the corpus or test coverage.
To put this framework into practice, create 25 to 50 representative cases, label required facts and relevant sources, define pass criteria, and run a baseline. Save both successful and failed outputs. Add automated checks for retrieval and formatting, route ambiguous cases to human reviewers, and publish a short report for every meaningful system change. That repeatable loop turns RAG evaluation from a one-time benchmark into a working part of AI application development.
