Prerequisites
- A working RAG pipeline (any retriever + any generator)
- 50-200 labeled Q&A pairs with expected answers and source docs
- Python 3.10+, access to a strong judge model (Claude 4, GPT-5, or similar)
Why this playbook
Most teams ship RAG with no evals. Then when quality degrades after an embedding model change, a retrieval tuning, or a vendor model update, they find out from users. This harness takes a few hours to build and saves you forever.
The three metrics that matter
- Faithfulness: does the generated answer only make claims supported by retrieved context? Catches hallucination.
- Answer relevance: does the answer actually address the question? Catches dodging.
- Context precision: of the retrieved chunks, how many were actually used / useful? Catches retrieval bloat.
These are enough to ship. More-exotic metrics (context recall, groundedness) are marginal gains.
Pro playbook
Unlock the full recipe
You've got the outcome and a free preview. Pro unlocks the complete prompt chain, failure modes, and every Pro playbook — for less than a coffee a week.
- Full multi-step prompt chains you can paste into Claude, GPT, or Cursor
- Failure-mode notes — what breaks in production and how to fix it
- Adjacent variations for cheaper models and tighter latency budgets
- New Pro playbooks as they ship — cancel anytime
Secure checkout via Stripe. No Keycloak required to buy — use the same email when you sign in later to access Pro. Cancel anytime from your account.
Already subscribed? to pick up where you left off.