What goes into an AI evaluation set
An AI pilot without an evaluation set is a demo. Here is how we build one in the first two weeks, what it should contain, and how to read the results.
The difference between an AI pilot that leads to a decision and one that leads to another meeting is usually the evaluation set. It is the list of real cases, agreed before anything is built, that the finished pilot is scored against.
This is how we assemble one during the assessment stage, before a line of code is written.
Why before, not after
If you collect test cases after the pilot exists, you will unconsciously collect the ones it handles well. Agreeing the set first keeps everyone honest, including us. It also turns "does it work?" from an opinion into a number.
How many cases
For a first pilot, a few dozen cases can support an initial assessment and a decision about whether to continue; the appropriate size depends on the workflow and on the consequences of an error. They are not enough to establish production reliability across every variation the workflow will meet; that takes a larger, growing set once the workflow is live and cases accumulate on their own. Too few and one unusual document swings the result. Too many and the review effort starts to outweigh the insight for a first pilot.
What the set must contain
A good set has two parts, and the report keeps them apart.
The first is a representative operational sample: cases drawn from a normal week in the proportions they actually occur. This part tells you what everyday workload the pilot would handle and is the only part that says anything about day-to-day effort or cost.
The second is a deliberately difficult challenge set, which over-weights the awkward cases because that is where the pilot will fail if it is going to. A result on the challenge set does not predict everyday performance or return on investment; it tells you where the edges are.
- Typical cases. The operational sample. The clean invoice, the clear enquiry, the well-formatted report, in their real proportions.
- Known difficult cases. Scans, handwriting, missing fields, two documents in one file, a message in a second language.
- Edge cases the team can name. Ask the people who do the work: "what is the one that always causes trouble?" They know.
- Cases that should be refused. A question the source documents cannot answer. An email that is not actually an enquiry. The correct output here is "I cannot do this; route to a person", and the set must include them so that behaviour is tested.
Each case has an agreed correct answer, written down by a person from your team. That takes time, usually a few hours spread over the assessment, and it is the most valuable few hours in the engagement.
How to read the results
We report four things together, in one table, because any one of them alone misleads.
- Accuracy on the cases that were answered.
- Error types. Not just how many were wrong, but how. A wrong total is a different problem from a wrong supplier name.
- Review rate. How often the system correctly said "I am not sure" and handed the case to a person. A high review rate on scans and a low one on typed documents is a good result, not a bad one.
- Cost per run. Model usage and hosting, so the running cost is visible before rollout.
The report shows counts and denominators alongside any rate, because accuracy on answered cases alone can hide extensive abstention: a system that answers ten cases correctly and sends ninety to review has 100% accuracy among answered cases and is not very useful. An illustrative layout, with fictional numbers:
| Case group | Cases assessed | Correct outcome, no edits | Sent to review as uncertain | Wrong | Cost per case |
|---|---|---|---|---|---|
| Operational sample: typed invoices | 40 | 36 | 3 | 1 | illustrative |
| Operational sample: scans | 20 | 9 | 10 | 1 | illustrative |
| Challenge set: credit notes, corrections | 10 | 4 | 5 | 1 | illustrative |
| Challenge set: should be refused | 6 | 6 correct refusals, routed to a person | n/a | 0 | illustrative |
All numbers in the table are illustrative. “Sent to review as uncertain” counts cases the system was unsure about. For the refusal group, a correct outcome means the system recognised the case as out of scope, declined it and routed it to a person; those cases are counted as correct outcomes, not as review, which is why that column reads n/a. Each wrong case is described in detail, because those are the ones that decide whether to roll out.
The decision
The evaluation ends with a recommendation: roll out, adjust and re-evaluate, or stop. Stopping is a valid outcome. It is far cheaper after a pilot of about 4 weeks than after a six-month rollout, and you keep the workflow map, the evaluation set and the findings either way.
If someone proposes an AI project without describing how it will be evaluated, that is the first question to ask. Building the set is part of the readiness assessment, and the report is the week-six deliverable described on the approach page.
Have a similar workflow?
Discuss what a focused first project could involve. Tell us which article you read.