AppliedGlossary
Evals
Evals are to LLM apps what unit tests are to code, with the same non-negotiable status. Without evals, you're guessing whether a new model, prompt, or RAG config is better. With evals, you ship with confidence.
A minimum viable eval: 50-100 labeled input/expected-output pairs, an automatic scorer (exact match, fuzzy match, LLM-as-judge, or human rubric), and a runner that produces a pass/fail score. Run before every prompt change, every model upgrade, every significant retrieval tweak. You can use a hosted eval platform or a short script of your own; the test cases matter more than the tool.
Example
# Minimal eval runner
import json
test_cases = json.loads(open("evals/support-tickets.jsonl").read())
total = correct = 0
for case in test_cases:
output = call_llm(case["input"])
score = llm_judge(case["expected"], output)
total += 1
correct += score
print(f"Pass: {correct}/{total} ({100*correct/total:.1f}%)")
# Ship-gating rule: no prompt change lands if eval score drops > 2 points.When to use it
- Any LLM feature that will ship; evals are the price of admission
- Comparing prompt variants or models
- Detecting regressions when a vendor updates their model
When not to use it
- Prototype stage: a ~10-case eval is fine, don't over-invest before product-market fit
- As a substitute for human review in high-stakes decisions
Articles that use this
- When to Use a Reasoning Model, and When Thinking Just Costs MoreTechniques
- MCP Server Design: Build Tool Interfaces That Agents Actually Use CorrectlyAgentic
- GPT-5.6 Pays You to Delete Your System Prompt: State Every Rule Once and Stop Repeating YourselfTechniques
- How to Prompt Reasoning Models: OpenAI's Seven-Part Prompt Structure, ExplainedTechniques
- Reasoning Effort: How to Set Thinking Depth Instead of Writing "Think Step by Step"Techniques

