Evals Explained

The shift in mindset

In traditional software, you write tests because behavior is deterministic. In AI software, you write evals because behavior is probabilistic — and that makes them more important, not less.

An eval is not a unit test. It's a measurement of system behavior on a fixed set of inputs.

What goes in an eval set

Three categories, balanced:

1. Sanity (easy) — basic cases the system must always handle. Tests for regression to known-broken states.
2. Capability (hard) — real-world tricky cases. Tests for whether your system can handle the actual job.
3. Adversarial (failure modes) — prompt injection, edge cases, things you've seen go wrong. Tests for safety + robustness.

Aim for 20–30 cases initially. You can always add more.

Evaluation methods, by task

LLM-as-judge — the practical pattern

For subjective tasks, use a strong model to score outputs against a rubric.

You are an expert reviewer. Score the response below on:
- Correctness (1-5)
- Helpfulness (1-5)
- Avoids hallucination (1-5)

Output JSON: { correctness, helpfulness, no_hallucination, justification }

Question: {{question}}
Expected answer key points: {{key_points}}
Actual response: {{response}}

Validate the judge against human ratings on 20 samples before trusting at scale. Drift happens.

Run evals in CI

The biggest impact: run evals on every change to prompts, agent code, or model versions. Treat eval score drops above N% as deploy blockers.

# .github/workflows/evals.yml
- name: Run evals
  run: python evals/run.py --threshold 0.85

Common pitfalls

• Evaluating on data the model was trained on — invalidates the eval. Always use fresh inputs.
• Eval set leaks into training — common with synthetic data. Keep eval set in a separate file, never in training pipelines.
• Optimizing the eval, not the system — when scores plateau, expand the eval set, don't just tweak it.
• Single-metric tunnel vision — track multiple dimensions (quality, latency, cost, safety) — improving one often regresses another.

Where to start tomorrow

1. Write 20 cases as JSON: { id, input, expected }.
2. Write a 50-line Python script that loops them, calls your system, scores them.
3. Run it. Fix the worst failures.
4. Add to the set as you discover failure modes.
5. Add it to CI when you have a baseline.

That's the whole game. Eval first, prompt-engineer second.