Building and evaluating a RAG system

Retrieval-augmented generation (RAG) is a pattern for answering questions with information stored outside a language model. The application searches a knowledge source, selects relevant passages, and gives them to a model as evidence for an answer.

The important part is that RAG is a system, not just a prompt. A fluent answer can still be wrong if retrieval found the wrong material, the source is stale, or the model claims more than the evidence supports.

The request flow

A useful mental model is:

Question → retrieve passages → assemble evidence → generate answer → cite sources

Ingest: read source files, extract text, and keep source metadata such as file name, page, and version.
Chunk: split long documents into passages small enough to retrieve and fit into the model context. Preserve headings and avoid cutting a definition in half.
Index: create searchable representations. Lexical search matches words; vector search can match meaning; hybrid retrieval combines both.
Retrieve: find candidate passages for the question. Apply access filters before returning evidence when documents have different permissions.
Rerank and assemble: rank the best candidates, remove duplicates, and build a prompt containing the question, selected evidence, and instructions for handling gaps.
Generate: ask the model to answer from the supplied evidence and return citations tied to the selected source chunks.
Inspect: log retrieval results, latency, errors, and model usage without exposing sensitive document text unnecessarily.

Chunking and retrieval choices

There is no universally correct chunk size. Small chunks make focused facts easier to find but can lose context. Large chunks preserve context but may add noise and use more tokens. Start with a simple, measured baseline, then compare alternatives using representative questions.

For every retrieved passage, retain enough metadata to trace it back to its source. A citation should identify a document and, where practical, a page, heading, or stable chunk reference. If content changes, re-index it and track the source version.

Evaluate retrieval before judging the model

Create a small evaluation set before tuning. For each question, record the expected source documents and, if possible, the facts a correct answer should contain. Include questions that have no answer in the knowledge base.

Useful retrieval measures include:

Recall@k: how often an expected source appears in the first k results.
Mean reciprocal rank (MRR): rewards putting the first relevant result near the top.
nDCG: accounts for relevance grades and ranking position when you have graded examples.

Then evaluate the answer separately: Is it correct? Is each claim supported by a retrieved passage? Do citations point to the right evidence? Does the system say it does not know when the documents do not answer the question?

A practical failure review asks:

Was the right document ingested and current?
Did chunking separate the needed fact from its context?
Did retrieval return a relevant passage?
Did reranking or context limits remove it?
Did the model misread the evidence or ignore the instruction?
Did the response cite the passage that actually supports the claim?

This separation helps you fix the right layer instead of changing the prompt to compensate for an indexing problem.

Keep the first version simple

Start with a no-model retrieval baseline. Verify that the correct evidence is found and show the passages to a reviewer. Add generation only after retrieval is measurable. Make failures visible: low confidence, missing evidence, invalid input, or model errors should lead to a safe fallback or human review.

For a small local implementation, the companion project uses TF-IDF retrieval and can optionally call a local Ollama model. It is a teaching example, not a production service; it has no authentication or access-control layer.