Without evals, every prompt change is a coin flip. With them, you can iterate confidently. Yet most teams skip them until something breaks — usually a regression that takes weeks to track down. Here's how to build evals before you regret not having them, and how to keep them useful as your system evolves.
What "eval" actually means
An eval is a test set plus a grader. For each example: an input, the desired output (or a set of acceptable answers), and a way to score the system's output. Run regularly, report aggregate scores, alert on regressions.
That's it. No magic. The hard part is building a test set that actually catches the failures that matter.
Three grading approaches
Exact match
The simplest. Output equals expected output? Pass. Works for classification, structured extraction, and tasks with canonical answers. Cheap, fast, deterministic.
Programmatic checks
Schema validation, regex matches, calling out to a reference function, parsing structured outputs and checking specific fields. The output is inspected programmatically.
This is usually the right tool. It's reliable, cheap, and fast. It also forces you to write down what "correct" means precisely — and that exercise alone catches a lot of fuzzy thinking.
LLM-as-judge
A different model grades the output against criteria. Useful for subjective quality (helpfulness, tone, completeness), but you need to evaluate the judge against human labels before trusting it.
Useful, but the slowest and most expensive option. Save it for cases where the others can't work.
Building your first eval set
50 examples beats 5,000 you never look at. Start with:
- The 20 things you most want the system to do well
- The 20 things you most want it to not do (refusals, edge cases, ambiguous inputs, adversarial inputs)
- 10 historical failures you've seen in dev or prod
Add to it whenever a regression surprises you in production. Every "wait, why did it do that?" moment should become an eval case.
The held-out set
Never iterate on the same examples you score against. If you tune a prompt against your eval set, you'll overfit to it and lose the signal. Split:
- Dev set — what you look at while iterating
- Eval set — what you score against, never look at directly
If you find yourself debugging specific eval failures, move those examples to the dev set and add new examples to the eval set. The eval set has to remain a clean signal.
Common pitfalls
- Vibes-only grading — eyeballing 5 outputs and shipping. Doesn't scale, doesn't catch drift, and is wildly biased toward whatever you wanted to be true.
- Overfitting to the test set — see above. Keep a held-out set.
- Trusting LLM-as-judge blindly — calibrate against human labels first. Judges have biases. Often they reward verbosity, sound-confident-but-wrong outputs, or specific phrasings.
- Letting evals rot — production usage shifts. Refresh your eval set quarterly. Old eval sets test capabilities you no longer care about.
- Grading too narrowly — an eval that only catches one failure mode misses the other twenty. Mix exact-match, structural, and judge-based grading.
Running evals in CI
Every prompt change triggers the eval suite. Score has to clear a threshold. Below it, the change is blocked.
This is the single highest-leverage discipline you can adopt for LLM apps. It costs maybe a day to set up the first time and saves dozens of incidents over a year.
Simple setup:
- Eval set lives in a JSON or YAML file in your repo
- A test script runs each example through your system, scores the output, prints aggregate stats
- CI runs the script on PRs; PR is blocked if pass rate drops below threshold
- Weekly job runs the same suite against production and alerts on regressions
Eval-driven prompt engineering
Most prompt iteration is a loop:
- Try a new prompt
- Check 3-5 examples by eye
- Ship if "looks better"
This is exactly how you regress everything subtly. Replace with:
- Try a new prompt
- Run the dev set, look at failures
- Run the held-out eval set, check pass rate
- Ship if pass rate improved without regressing critical examples
What to measure
Beyond pass rate, track:
- Per-category breakdown — "the system is 90% pass overall but 40% on policy refusals" tells you where to focus
- Confidence/uncertainty calibration — does the system know when it doesn't know?
- Latency and cost per example — quality at 10x cost isn't a win
- Regression rate — what percent of previously-passing examples now fail?
The mindset shift
Evals turn LLM development from "ship and hope" into engineering. They're the most underrated investment in AI applications. Build them early, refresh them often, and you'll iterate faster than teams that don't — by a lot.
If you want a hands-on tour of building evals for production RAG and agents, the JoinAI MasterClass dedicates a full week to it.
