RAG Tutorial: Build, Evaluate, and Maintain a Retrieval-Augmented Generation App

Overview

Retrieval-augmented generation (RAG) combines a language model with a searchable collection of external content. Instead of asking a model to answer only from its training or conversation context, the application retrieves relevant passages at query time and supplies them to the model as evidence.

A typical RAG application has six stages:

1. Ingestion: collect source files, web pages, records, or other approved content.
2. Preparation: extract text, remove unusable material, preserve metadata, and divide documents into chunks.
3. Embedding: convert chunks into numerical representations that support semantic search.
4. Indexing: store embeddings, text, and metadata in a vector database or another retrieval system.
5. Retrieval: find the most relevant passages for a user question, optionally combining semantic and keyword search.
6. Generation: provide the retrieved context to the model and require an answer that follows your grounding and citation rules.

RAG is not simply a database lookup added to a chatbot. The quality of the final response depends on the entire chain. Poor extraction can make good documents unusable. Oversized chunks can dilute relevance, while tiny chunks can remove important context. A strong model cannot reliably compensate for missing or incorrectly retrieved evidence.

Before choosing a vector database or model, define the application’s job. Specify which questions it should answer, which sources are authoritative, what it should do when evidence is missing, and which users may access each source. For a broader production workflow, see this RAG application tutorial.

Checklist by scenario

Scenario 1: Building a first RAG prototype

• Write down three to five representative user questions before building the pipeline.
• Choose a small, clearly owned document set rather than importing every available file.
• Record source metadata such as title, URL, document type, owner, version, and update date.
• Test text extraction on PDFs, tables, headings, lists, and scanned pages separately. Do not assume every format produces clean text.
• Start with a simple chunking rule, then inspect the actual chunks manually.
• Store the original text alongside each embedding so retrieved evidence can be displayed and audited.
• Use a prompt that tells the model to answer from the supplied context, identify uncertainty, and avoid inventing unsupported details.
• Return source references with every answer where practical, even if the first interface is internal.

A prototype should prove that the right passages can be found for real questions. It does not need every operational feature on day one, but it should expose the retrieved chunks and the final prompt during testing. That visibility makes errors easier to diagnose.

Scenario 2: Improving retrieval quality

• Build a small evaluation set containing questions, expected source documents, and notes about the required answer.
• Compare retrieved passages against the expected evidence, not only against the model’s final wording.
• Try chunk boundaries based on headings, paragraphs, sections, or records instead of relying only on character count.
• Preserve document titles and section headings in the indexed text so a chunk retains navigational context.
• Add metadata filters for department, product, region, date, permission group, or document status where those distinctions matter.
• Test hybrid retrieval when exact identifiers, error codes, product names, or legal phrases are important.
• Consider reranking retrieved candidates before sending them to the language model.
• Inspect failed queries separately: some failures come from retrieval, while others come from an incomplete source collection.

Do not measure success only by whether an answer sounds plausible. A fluent answer can still be based on the wrong document. Keep retrieval tests and generation tests separate so you know which part of the system needs attention. The RAG evaluation guide provides a useful framework for organizing these checks.

Scenario 3: Preparing for production use

• Define which users, teams, or services may retrieve each document.
• Apply authorization filters before or during retrieval, not only after the model has received the context.
• Decide how deleted, expired, superseded, or restricted documents are removed from the index.
• Log query identifiers, retrieval results, model inputs, latency, errors, and citations without unnecessarily storing sensitive content.
• Set limits for input size, retrieved context, response length, and request frequency.
• Provide a clear fallback when no relevant evidence is found: ask a clarifying question, return a controlled refusal, or route the request elsewhere.
• Test prompt injection in source documents and user queries. Treat retrieved text as untrusted data, not as instructions.
• Use a stable prompt and model configuration during evaluation so changes can be compared fairly.

Security and permissions are part of the RAG architecture, not later interface features. A response can be factually correct and still be an access-control failure if it reveals content the requesting user should not see. Review the prompt injection prevention guide before exposing retrieved content to untrusted inputs.

Scenario 4: Maintaining an existing index

• Track the source version or checksum used to create each indexed record.
• Use incremental updates when possible, but periodically verify that the index matches the source repository.
• Remove stale embeddings when documents are deleted or replaced.
• Reprocess files when extraction, chunking, embedding, or metadata rules change.
• Keep a record of which pipeline version created each index.
• Run a fixed evaluation set after changes to source content, retrieval settings, prompts, or models.
• Review user feedback for recurring missing sources, incorrect citations, and unanswered questions.

What to double-check

Chunking and metadata

There is no universal chunk size that works for every corpus. A policy manual, source-code repository, product catalog, and support transcript have different structural needs. Begin with boundaries that preserve meaning, then inspect whether each chunk can answer a question without requiring excessive surrounding text. Include enough metadata to identify the source and apply access rules, but avoid duplicating irrelevant material across every chunk.

Retrieval and context assembly

Check whether the retriever returns relevant passages near the top of the result list. Also check for redundancy: several nearly identical chunks can crowd out complementary evidence. Context assembly should preserve source boundaries and make citations traceable. If the application combines multiple sources, label them clearly and decide how conflicting versions are handled.

Grounding and citations

Prompt instructions should distinguish between evidence and instructions. A practical policy is to require the model to answer only when the retrieved material supports the claim, identify gaps, and cite the specific source passages used. Test questions with incomplete, conflicting, and deliberately misleading context. The goal is not merely a confident response; it is a response whose important claims can be checked.

Evaluation scorecard

Keep a lightweight scorecard with the following fields:

• Question: the user request being tested.
• Expected evidence: the document or passage that should be retrieved.
• Retrieval result: whether the required evidence appeared in the candidate set.
• Grounding: whether the answer stayed within the supplied evidence.
• Completeness: whether it addressed the important parts of the question.
• Citation accuracy: whether references support the claims they are attached to.
• Access result: whether restricted content was correctly excluded.
• Operational notes: latency, failure mode, and any unusual behavior.

Use the same scorecard when comparing a new embedding model, chunking strategy, retriever, prompt, or language model. This makes prompt optimization and other changes easier to evaluate systematically; the prompt optimization framework can help structure those comparisons.

Common mistakes

• Indexing everything without ownership: Unreviewed, duplicate, or obsolete sources can reduce trust. Start with a defined collection and clear owners.
• Assuming embeddings solve exact-match search: Identifiers, codes, names, and quoted language may need keyword or hybrid retrieval.
• Ignoring document structure: Flattened text can separate headings from the content they explain and damage table meaning.
• Sending too much context: More retrieved text is not automatically better. Remove redundancy and prioritize evidence relevant to the question.
• Evaluating only final answers: Inspect retrieval independently so generation errors are not confused with search failures.
• Trusting citations without checking them: A source label is useful only when it points to evidence that supports the claim.
• Applying permissions too late: Never retrieve restricted passages merely to hide them in the final interface.
• Updating documents without updating the index: Stale embeddings can produce answers that appear reasonable but describe an old state.
• Changing several variables at once: If the model, prompt, chunking, and retriever all change together, performance differences are difficult to explain.

When to revisit

Revisit this RAG checklist whenever the underlying inputs or workflow change. A scheduled review before seasonal planning cycles is useful when documents, policies, catalogs, or support content are updated in batches. Also review the system after a major product release, repository migration, permission change, model change, or retrieval configuration change.

At each review, sample recently updated documents, confirm that deleted material no longer appears, rerun the evaluation set, and inspect a selection of citations. Compare retrieval failures with user feedback rather than relying on a single quality score. If response time has changed, review the number of retrieved passages, reranking work, model selection, caching, and logging overhead; the LLM latency checklist covers these operational considerations.

For a practical next step, choose a narrow document set and create ten representative questions. Record the expected evidence, inspect the retrieved chunks, and score the answers for grounding, completeness, citations, and permissions. Fix the largest repeatable failure first. Then preserve the test set as a baseline for every future change to your RAG application.