RAG Application Tutorial: Build a Production-Ready Retrieval-Augmented Generation Workflow

Overview

Retrieval-augmented generation, or RAG, connects an LLM to a searchable collection of external information. Instead of asking a model to answer from its general training alone, the application retrieves relevant passages and places them in the prompt supplied to the model. The model then produces an answer grounded in that retrieved context.

A production-ready RAG application is more than a vector database and an API call. It is a chain of decisions:

1. Collect and normalize source documents.
2. Split documents into retrievable chunks while preserving useful context.
3. Generate embeddings and store them with metadata.
4. Retrieve and optionally rerank candidate passages.
5. Construct a controlled prompt for the generation model.
6. Return an answer with citations or source references.
7. Measure retrieval, answer quality, latency, cost, and failure rates.

The main planning mistake is estimating only generation cost. A useful RAG estimate includes ingestion, embedding, storage, retrieval, reranking, generation, observability, and engineering overhead. It should also distinguish fixed work, such as indexing a document collection, from variable work, such as handling each user query.

For a deeper quality framework, see the RAG Evaluation Guide. RAG is also one option among prompt engineering, fine-tuning, and other approaches; compare the tradeoffs in Fine-Tuning vs Prompt Engineering vs RAG.

How to estimate

Start with a simple monthly model. Replace each input with a value from your own system or a clearly labeled planning assumption.

Monthly RAG cost = ingestion cost + embedding cost + storage cost + query cost + monitoring cost + maintenance cost.

For query cost, use:

Query cost = monthly queries × cost per query.

A more detailed query estimate is:

Cost per query = retrieval infrastructure cost per query + reranking cost per query + input-token cost + output-token cost.

The token component can be estimated as:

Token cost per query = (input tokens ÷ 1,000,000 × input rate) + (output tokens ÷ 1,000,000 × output rate).

Use the actual provider pricing and billing units when you fill in the rates. Prices, model limits, included quotas, and storage terms can change, so keep them in a configuration sheet rather than hard-coding them into a written estimate.

Cost is only one part of the decision. Track expected outcomes with a small evaluation set containing representative questions and known source passages. Useful measures include:

• Retrieval recall: whether the relevant passage appears among the retrieved results.
• Ranking quality: whether the most useful passages appear near the top.
• Groundedness: whether the answer is supported by the supplied context.
• Answer completeness: whether the response addresses the important parts of the question.
• Citation correctness: whether each citation actually supports the associated claim.
• Operational performance: latency, errors, timeouts, and cost per successful answer.

Do not optimize a single metric in isolation. Increasing the number of retrieved chunks may improve recall while increasing prompt length, latency, and the chance of distracting the generation model. The best configuration is usually the least expensive one that meets the required quality and reliability thresholds.

Inputs and assumptions

Build an input table before choosing infrastructure. Record the value, unit, source, and date for each assumption.

Corpus inputs

• Number of source files, pages, records, or web documents.
• Average and maximum document length.
• File types and extraction requirements.
• Update frequency and expected monthly change volume.
• Access rules, tenant boundaries, and document retention requirements.

Chunking and indexing inputs

• Target chunk size and overlap.
• Whether headings, tables, lists, and page references are preserved.
• Embedding model, vector dimensions, and reindexing policy.
• Metadata fields such as document ID, section, date, owner, and permissions.

Chunk size should follow the structure of the information rather than a universal number. A policy clause, product specification, or troubleshooting step may be a useful unit on its own. Overlap can protect continuity, but excessive overlap creates duplicates and increases storage and retrieval noise.

Query and generation inputs

• Monthly query volume and peak requests per minute.
• Expected retrieved passage count and average passage length.
• Generation model, maximum input context, and output length.
• Percentage of queries requiring reranking, filters, or conversational history.
• Target response time and acceptable fallback behavior.

Include security assumptions from the beginning. Treat retrieved documents as untrusted input, enforce authorization before retrieval, and avoid allowing document text to override application instructions. The Prompt Injection Prevention guide covers relevant controls. For latency planning, use the LLM Latency Optimization Checklist.

Worked examples

Example 1: Internal documentation assistant

Assume a team has a stable collection of internal guides and receives 20,000 questions per month. The planning sheet records one initial indexing event, a smaller monthly update batch, a fixed vector-storage estimate, and a variable query estimate based on retrieved context and generated answer length.

The calculation is:

Monthly estimate = monthly reindexing and embedding work + storage + (20,000 × variable cost per query) + monitoring.

Run the same calculation for two configurations: a lightweight retrieval path and a higher-quality path that adds reranking or a more capable generation model. Compare not only total cost, but also evaluation results for permissions, citation correctness, and questions involving multiple documents. If the higher-cost path does not improve the required measures, the simpler design is easier to operate.

Example 2: Frequently changing knowledge base

Now assume a support knowledge base changes every day. The initial corpus size may be modest, but the update rate is high. In this case, estimate incremental ingestion rather than treating every update as a full rebuild:

Monthly embedding work = number of changed documents × average chunks per document × embedding cost per chunk.

Test whether changed documents can be identified reliably and whether obsolete chunks are deleted or marked inactive. A stale result can be more damaging than a missing result, so freshness belongs in the evaluation set. Include questions whose answers changed recently and verify that citations point to the current version.

Example 3: Cost sensitivity check

Calculate a low, expected, and high scenario by changing only a few variables: query volume, retrieved context size, output length, update frequency, and model choice. This gives you a range rather than a false-precision single number. Also calculate the effect of caching repeated questions, limiting unnecessary conversation history, reducing irrelevant retrieved chunks, or routing simple questions to a less expensive model. These changes should be validated against answer quality rather than applied solely to reduce spend.

When to recalculate

Revisit the estimate whenever a major input changes. At minimum, recalculate after:

• A model, embedding service, vector store, or reranker is changed.
• Provider prices, quotas, billing units, or context limits change.
• Query volume or peak traffic moves beyond the planning range.
• The document collection grows, changes format, or updates more frequently.
• Chunking, metadata filters, retrieval depth, or prompt structure is modified.
• Evaluation quality, latency, error rate, or citation behavior shifts.
• New tenants, permission rules, retention requirements, or compliance controls are introduced.

Use this troubleshooting decision tree when results deteriorate:

1. No relevant source is retrieved: inspect document extraction, chunk boundaries, metadata filters, query rewriting, and embedding coverage.
2. Relevant sources are retrieved but ranked poorly: test retrieval depth, hybrid search, metadata weighting, or a reranker.
3. The answer ignores relevant context: simplify the prompt, reduce noisy passages, make source boundaries clear, and test the model's context handling.
4. The answer is unsupported: require citations, add an abstention rule, and evaluate claims against retrieved passages.
5. Latency or cost is too high: measure each pipeline stage, then test caching, batching, smaller context, model routing, or asynchronous ingestion.
6. Permissions fail: stop retrieval before generation, review tenant and document-level filters, and add authorization cases to the evaluation set.

Keep a versioned record of chunking settings, prompts, retrieval parameters, model identifiers, pricing inputs, and evaluation results. Prompt changes should be reviewable and reversible; see Prompt Versioning Best Practices. Before launch, use this checklist: define the corpus, preserve metadata, test extraction, create a representative evaluation set, measure retrieval and grounded answers, enforce permissions, add citations, instrument cost and latency, and schedule the next review. Recalculate after real usage data replaces planning assumptions.