Skip to content
Search
DevelopmentGlossary

Prompt Regression Testing

A prompt change that fixes one case often breaks another. Regression testing makes that visible. You keep a set of inputs with expected results (a golden dataset), run it on the current version and on the candidate, score both the same way, and block the change if the score drops or if any must-pass case fails. It's the release gate side of evals.

Score with exact checks where you can (valid JSON, the right label, required fields present) and use LLM-as-judge rubrics for open-ended text. Pin the model version in your config so a provider update shows up as a deliberate change you can test. Run the suite in CI next to your unit tests. The playbook build a RAG eval harness that catches regressions walks through a full harness, and Qurtoo's article on writing grading rubrics covers the scoring side.

Example

# prompt-regression.yaml
suite: support-replies
dataset: evals/support-replies.golden.jsonl   # 120 cases, 15 marked must_pass
compare: [prompts/support-v12.md, prompts/support-v13.md]
model: pinned-model-id
scorers:
  - valid_json
  - contains_required_fields: [answer, next_step]
  - judge_rubric: rubrics/support-quality.md
block_release_if:
  - any must_pass case fails
  - average judge score drops by more than 2 points

When to use it

  • Any prompt that runs in production
  • Before switching or upgrading models
  • Changes to retrieval, chunking or tools that feed the prompt

When not to use it

  • Early exploration, when the task itself is still changing every day
  • As a replacement for looking at real outputs now and then