How to Evaluate a RAG Pipeline

Without evaluation, every change to chunking, embeddings or prompts is a guess. A small labelled dataset turns RAG tuning into engineering.

1. Build an evaluation set

• Collect 50-200 real questions from logs, support tickets or users.
• For each, record the ID(s) of the chunk or document that contains the answer, and a reference answer.
• Include questions the docs can't answer, to test that the system says so.

2. Measure retrieval separately

def recall_at_k(eval_set, search, k=5):
    hits = 0
    for item in eval_set:
        found = {r["doc_id"] for r in search(item["question"], limit=k)}
        hits += bool(found & set(item["relevant_doc_ids"]))
    return hits / len(eval_set)

def mrr(eval_set, search, k=10):
    total = 0
    for item in eval_set:
        for rank, r in enumerate(search(item["question"], limit=k), start=1):
            if r["doc_id"] in item["relevant_doc_ids"]:
                total += 1 / rank
                break
    return total / len(eval_set)

If recall@k is low, no prompt will save you. Fix retrieval first.

3. Measure the answers

• Correctness - does it match the reference answer?
• Faithfulness - is every claim supported by the retrieved context?
• Refusal accuracy - does it decline when the answer isn't in the docs?

LLM as a judge

JUDGE = """Compare the answer to the reference.
Reply with JSON: {"correct": true|false, "unsupported_claims": [..]}.

<question>{q}</question>
<context>{ctx}</context>
<reference>{ref}</reference>
<answer>{ans}</answer>"""

Use structured outputs for the judge's verdict, and spot-check a sample by hand to make sure the judge agrees with you.

4. Track it over time

Run the eval in CI or before each deploy, and keep a simple table of scores per change. Small, measured improvements add up quickly.