What is an AI eval dataset?

An AI eval dataset is a curated set of representative inputs, each paired with an expected output or a scoring rule, that you run a prompt or model against to measure quality objectively — the AI equivalent of a test suite. Instead of eyeballing a few examples after a change, you run the whole set and get a score, so 'it feels better' becomes 'it scored higher.' A good eval set covers the normal cases, the edge cases, and the specific failures that have burned you before — every production incident becomes a new eval case, so it can never silently recur. It's the single practice that most separates an AI capability you can change confidently from one nobody dares touch.

How do you score AI outputs in an eval?

By the strictest method the task allows. Exact-match or rule-based scoring is best where the output is structured or has a right answer (a field value, a classification, valid JSON) — it's cheap, deterministic and unarguable. For open-ended outputs, you fall back to criteria-based scoring: a rubric of what 'good' means, applied either by a human on a sample or by a model-as-judge (another model scoring against your rubric), each with trade-offs. The rule of thumb is to make as much of the output rule-checkable as you can — structure it so parts can be scored exactly — and reserve subjective scoring for the genuinely open parts, because deterministic scoring is the part of an eval you can actually trust without second-guessing.

Enterprise AI Evaluation & Regression Testing

Q: What is regression testing for AI?

Regression testing for AI is running your eval set on every change — a prompt edit, a settings change, a new model version — and comparing the score to the previous baseline, so a change that quietly breaks something is caught before it ships. It works exactly like code regression tests: you have a known-good baseline, and a change that drops the score on cases that used to pass is a regression to investigate, not a release to push. The gate is the discipline: a change that regresses the eval doesn't ship until someone understands why. Without it, the people downstream are your regression suite, and a 'better' prompt that broke three edge cases reaches them before you notice.

"It feels better" is how AI features quietly get worse. Someone tweaks a prompt, tries two or three examples, they look good, it ships — and the behaviour has just changed for every input nobody tried, some of it for the worse. That is the core reliability problem of building with models: the output space is huge, and eyeballing a handful of cases tells you almost nothing about the rest. Inside an enterprise it's also a governance problem, because "it looked better to me" is not evidence anyone can approve a change on. Evals are the answer, and they're the practice that most separates an AI capability a team can improve confidently from one nobody dares touch. Prompt and model versioning gives you the safe way to change things; evals are how you know a change was actually an improvement. This is how I build them.

Evals are a test suite for a fuzzy function

The reframe that makes evals click: an AI step is a function with a huge, fuzzy output space, and an eval set is its test suite. Same idea as code tests — known inputs, expected results, run on every change — adapted to the fact that "expected" is sometimes a rule and sometimes a judgement.

Once you hold that framing, the whole practice follows: build a representative set, define how to score it, run it on every change, and gate on the result. The payoff is the same as tests give code — the confidence to change things without breaking what worked, and something concrete to show whoever has to sign the change off.

Building the set: cover three kinds of case

An eval set is only as good as its coverage. Three kinds of case earn their place:

Case type	What it is	Why it matters
Normal	The typical, happy-path inputs	The bulk of real volume — must stay good
Edge	Boundary, unusual-but-valid, tricky inputs	Where quality quietly degrades first
Regression	The exact inputs that have failed before	So a fixed bug can never silently come back

The regression cases are the ones people skip and the ones that pay off most: every incident becomes a new eval case. A bad output someone reports isn't just fixed — it's added to the set, so any future change that would reintroduce it fails the eval. Over time the set becomes a memory of every way the capability has been wrong, and a standing guarantee it won't be that way again silently. That is also the most useful thing an incident review can produce: not an action item, a case.

Two practical notes. Cases should come from real volume rather than from what someone imagined in a workshop — the distribution of odd inputs is never what people expect. And the set inherits the classification of the data in it: if the cases carry real customer or transaction data, the eval set is production data and has to live where that data is allowed to live.

Turn every incident into an eval case. Do that consistently and your eval set stops being a test suite and becomes institutional memory — a growing proof that the capability can't regress in any way it's failed before.

Scoring: rule-based where you can, judgement where you must

Scoring is where evals live or die, and the rule is simple: make as much rule-checkable as the task allows.

Exact / rule-based — for structured or right-answer outputs (a field, a classification, valid JSON against a schema). Cheap, deterministic, unarguable. This is the scoring you can fully trust.
Criteria / rubric — for open-ended outputs, a defined rubric of what "good" means, applied by a person on a sample or by a model-as-judge (another model scoring against your rubric). More coverage, but more subjective — treat its numbers as directional, not absolute.

The design move that makes evals more reliable is structuring the output so more of it is rule-checkable — the same instinct as an output contract. If part of the response is a structured field, you can score that part exactly and reserve the fuzzy scoring for the genuinely open part. Deterministic scoring is the bedrock; build as much of your eval on it as the task permits.

The regression gate: run on every change

An eval set that isn't run automatically is a good intention. The discipline is a gate: every change — prompt, settings, model version — runs the set against a known-good baseline, and a change that drops the score on cases that used to pass is a regression to investigate, not a release to ship.

This is exactly a code regression suite, applied to the model layer:

Baseline. The current eval score is the bar.
Change. Edit the prompt, or adopt a new model version.
Run. Score the whole set.
Compare. Improved, held, or regressed?
Gate. A regression doesn't ship until someone understands it.

Step 5 is the point. Without the gate, a "better" prompt that broke three edge cases reaches live before anyone notices — and the people downstream become your regression suite, reporting failures as rework. Attach the score to the change record while you're at it: it turns an approval from a judgement about the person making the change into a judgement about evidence. It's also what makes adopting a new model version safe — you don't trust "it's newer, it's better," you run your evals and see whether it's better on your tasks.

And the same evidence answers a bigger question. The measured exception rate on a step — how often it's wrong, and whether the wrongness is detectable — is what tells you whether that step is a candidate for automation, augmentation or autonomy. Without evals, that decision is an assertion.

Keep it proportionate

None of this needs a platform, and the first version shouldn't be one. An eval set is a folder of cases in version control and a job that scores them — start with ten cases that matter and grow it every time something breaks. It's the same instinct as measuring only the few things that change a decision: the mistake isn't a small eval set, it's having none and shipping prompt changes on vibes. Ten real cases scored on every change beats a hundred nobody runs, and it beats a tooling selection that takes two quarters.

What I would decide

Build the eval set from the start, small, and grow it from real failures — every incident becomes a permanent case. Score as much as possible with rules, and structure outputs so you can; reserve subjective scoring for the genuinely open parts and treat its numbers as directional. Gate every change on the set, keep the score with the change record, and never let a regression ship unexamined — that gate is the whole value, because it's what lets a team change the capability without the people downstream discovering the breakage first. Evals turn "I think this is better" into "this scored better," and that single shift is what makes an AI capability something an organisation can improve with confidence instead of fear.