Build a RAG Eval Harness That Catches Regressions

Before you start

• A working RAG pipeline (any retriever + any generator)
• 50-200 labeled Q&A pairs with expected answers and source docs
• Python 3.10+, access to a strong judge model (Claude 4, GPT-5, or similar)

Why this playbook

Most teams ship RAG with no evals. Then when quality degrades after an embedding model change, a retrieval tuning, or a vendor model update, they find out from users. This harness takes a few hours to build and saves you forever.

The three metrics that matter

1. Faithfulness: does the generated answer only make claims supported by retrieved context? Catches hallucination.
2. Answer relevance: does the answer actually address the question? Catches dodging.
3. Context precision: of the retrieved chunks, how many were actually used / useful? Catches retrieval bloat.

These are enough to ship. More-exotic metrics (context recall, groundedness) are marginal gains.

Step 1: Assemble your eval set

Minimum 50 pairs, target 200. Pull from real user queries if you have them. Each entry has:

{
  "id": "q001",
  "question": "What is the return window for electronics?",
  "expected_answer": "45 days for electronics; 30 days for non-electronics.",
  "expected_sources": ["electronics-policy.md", "returns-policy.md"],
  "hardness": "easy" | "medium" | "hard",
  "tags": ["policy", "returns"]
}

Step 2: Evaluator functions

Use LLM-as-judge for each metric. Each function takes the question, retrieved context, and generated answer, returns a 0.0-1.0 score.

def faithfulness(question, context, answer, judge_model="claude-opus-4-6"):
    prompt = f"""Given the question, the retrieved context, and an answer,
identify every factual claim in the answer and check whether each is
supported by the context. Return JSON: {{"claims": [{{"claim": str, "supported": bool}}], "score": float 0-1}}

Question: {question}
Context: {context}
Answer: {answer}"""
    return call_judge(judge_model, prompt).parsed["score"]

def relevance(question, answer, judge_model="claude-opus-4-6"):
    prompt = f"""Does the answer directly address the question?
Score 0 (completely off-topic) to 1 (fully addresses).
Return JSON: {{"score": float, "reason": str}}

Question: {question}
Answer: {answer}"""
    return call_judge(judge_model, prompt).parsed["score"]

def context_precision(retrieved_chunks, question, answer, judge_model="claude-opus-4-6"):
    useful = 0
    for chunk in retrieved_chunks:
        prompt = f"""Did this chunk meaningfully contribute to answering the question?
Answer YES or NO.
Question: {question}
Chunk: {chunk}
Answer: {answer}"""
        if "YES" in call_judge(judge_model, prompt).text.upper():
            useful += 1
    return useful / max(1, len(retrieved_chunks))

Step 3: Runner

import json
from statistics import mean

def run_evals(eval_set: list[dict], rag_pipeline):
    results = []
    for e in eval_set:
        chunks = rag_pipeline.retrieve(e["question"])
        answer = rag_pipeline.generate(e["question"], chunks)
        ctx_str = "\n\n".join(c.text for c in chunks)
        results.append({
            "id": e["id"],
            "faithfulness": faithfulness(e["question"], ctx_str, answer),
            "relevance": relevance(e["question"], answer),
            "context_precision": context_precision(chunks, e["question"], answer),
        })
    return {
        "per_case": results,
        "aggregate": {
            "faithfulness": mean(r["faithfulness"] for r in results),
            "relevance": mean(r["relevance"] for r in results),
            "context_precision": mean(r["context_precision"] for r in results),
        },
    }

Step 4: Ship gates

Define thresholds you won't ship below. Start permissive, ratchet up as you improve.

MIN_THRESHOLDS = {
  "faithfulness": 0.85,
  "relevance": 0.80,
  "context_precision": 0.50,  # deliberately lenient at start
}

def ship_gate(results):
    failures = []
    for metric, threshold in MIN_THRESHOLDS.items():
        score = results["aggregate"][metric]
        if score < threshold:
            failures.append(f"{metric}: {score:.3f} < {threshold}")
    if failures:
        raise AssertionError("Ship gate failed:\\n" + "\\n".join(failures))

Step 5: CI wiring

# .github/workflows/rag-evals.yml
name: RAG evals
on:
  pull_request: { paths: ["rag/**", "prompts/**"] }

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: python scripts/run_evals.py  # writes eval-output.json, then calls ship_gate(); a failed gate exits non-zero and fails the job
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: eval-results
          path: eval-output.json

Step 6: Keep the eval set fresh

• Add 5-10 new hard cases each month, pulled from real user queries that the system got wrong
• Rotate out cases that all variants pass -- they're no longer signal
• Tag hard cases by category so you know where you're weakest

Failure modes

• Judge bias: the same judge consistently over- or under-scores. Validate with a human spot-check on 10 cases per run.
• Eval set drift: early eval cases stop reflecting real usage. Refresh quarterly.
• False positives in context precision: judge calls a chunk useful because it LOOKS relevant even if the answer didn't use it. Cross-validate by checking whether the final answer quotes the chunk.

Variations

• Pairwise comparison evals: instead of absolute scores, judge which of two RAG variants answers better. Useful for A/B decisions.
• Hallucination detection as a classifier: train a small classifier on your own faithfulness-judged data for cheap inline detection at inference time.
• Multi-turn evals: extend to conversational Q&A where context carries across turns.