What an evaluation owes you

An evaluation is the part of an AI project where the money is actually decided, and it is almost always the part that gets the least scrutiny. Somebody shows you ninety-four percent, the room nods, and the procurement process begins. Nobody asks ninety-four percent of what, measured on which examples, against what a person doing the same task would have scored, which are the three questions that determine whether the number means anything at all.

This is not a complaint about vendors. The same thing happens in internal projects, built by careful people, because an evaluation is genuinely hard to design and a demo is easy to produce. The difference between them is not effort or honesty; it is that an evaluation is built to be able to fail and a demo is built to be shown. A number that could not have come out badly is not a measurement, and recognising that distinction is most of the skill here.

A number needs a number beside it

The single most useful habit in this area is to refuse to read an accuracy figure that arrives alone. Ninety-four percent is excellent for some tasks and catastrophic for others, and which one it is depends entirely on what the alternatives score. If the task is classifying support tickets into twelve categories and the most common category accounts for ninety-one percent of them, then a system that always guesses that one category scores ninety-one, and the model has bought you three points.

That comparison is called a majority-class baseline and computing it takes about a minute. We ask for it in every evaluation we review, and in roughly a third of the cases it turns out that nobody had computed it, and in a meaningful fraction of those the headline number stops being impressive the moment it appears. This is not usually an attempt to mislead. It is that the baseline is boring to compute and nobody asked.

There is a second baseline worth the same minute, and it is the one that decides the business case rather than the technical one: what does the current process score? Not the theoretical best case, the actual current process, including the Tuesday when two people were on holiday. A model that is slightly worse than a careful human and substantially better than the real queue on a bad week is a perfectly good investment, and you cannot see that without measuring the real queue.

Where the test examples came from

The second question is where the evaluation set came from, and the answer you are listening for is that somebody chose it before they knew how the system performed. A test set assembled after the fact, by picking examples from a pool, is not a test set — it is a selection, and the selection pressure runs in exactly the direction that makes the number look good, whether or not anybody intended that.

The practical version of this question is: can I see the examples? Not a summary, the actual rows. Reading thirty of them tells you more about the evaluation than any metric, because you can see whether they look like the work. In one review we found that every example in a 400-row evaluation set had come from a single month, and that month was the month before a product change that altered the vocabulary customers used.

The leakage question

If the system was trained or tuned on anything, the next question is whether any evaluation example could have been seen during that process. This is the most common way an honest team produces an inflated number, because the leak is usually indirect.

• The same ticket appearing twice under different IDs.
• A near-duplicate from a template.
• A date field that lets the model infer the answer.

The check is not sophisticated — look for duplicates, then look for near-duplicates — and it is skipped far more often than it is performed.

One number hides the failures

Aggregate accuracy is an average, and averages hide distributions in this domain exactly as they do in performance work. A system at ninety-four percent overall can be at ninety-nine on the eight common categories and forty on the four rare ones, and whether that is acceptable depends on something no metric knows: whether the rare categories are the cheap ones or the ones involving a refund, a legal obligation or a safety issue.

Ask for the breakdown by class and read the worst row first. Rare and expensive is the dangerous combination, because rarity means the aggregate metric will never report the problem and expense means each instance matters. This is also the reason a confusion matrix is worth more than an accuracy figure: it tells you not just how often the system is wrong but what it is wrong in the direction of, and those two mistakes usually have very different costs.

The cost of each kind of mistake

Most real decisions turn on asymmetry. Approving a fraudulent transaction and declining a legitimate one are both errors and they are not remotely the same error, so a system tuned to minimise total errors will be tuned wrongly for the business. The useful artefact here is a short table giving the cost of each mistake type in whatever unit you actually care about — money, hours, a complaint — and it is a product decision rather than a technical one.

Once that table exists, the threshold stops being an engineering preference and becomes arithmetic. It also makes the evaluation legible to people who do not read metrics, which matters more than it sounds: the person who has to live with the false positives is usually not in the technical review, and giving them a number in their own units is how they get to object before the system ships rather than afterwards.

Run the same input twice

For generative systems there is an additional question that classical evaluation never had to ask: is the output stable? The same prompt can produce different answers, which means a single-run evaluation measures one sample from a distribution and reports it as the system’s behaviour. Running the evaluation set three times and reporting the spread is more honest and frequently more surprising than the headline, particularly on the examples near a decision boundary.

This also changes what you can promise an operations team. A system that is right ninety-four percent of the time and gives the same answer every time is operationally quite different from one that is right ninety-four percent of the time and occasionally changes its mind about yesterday’s case, because the second one generates support conversations that are very hard to resolve.

Who scored the answers

For any task where correctness is a judgement rather than a lookup, somebody had to decide what the right answer was, and that person is part of the measurement. If one engineer labelled all the examples, the evaluation measures agreement with that engineer. The standard remedy is to have two people label an overlapping sample and report how often they agree, because a task where two careful humans agree only eighty percent of the time cannot sensibly be evaluated against a ninety-five percent target.

This number — inter-annotator agreement — is also the ceiling. It tells you the best score any system could honestly achieve on this task as defined, and when it is low the correct response is usually to sharpen the definition rather than to buy a better model. We have seen a project spend two months on model selection for a task whose labels two of its own experts disagreed on a fifth of the time.

When the model does the scoring

Using a model to grade another model’s output is now common and it is useful, but it needs its own validation: somebody has to check the grader against human judgement on a sample, and report that agreement alongside the result. Without it the evaluation is a measurement of how much two models resemble each other, which is a real quantity and not the one anybody wanted.

An evaluation is not a monitor

The evaluation tells you how the system performed on a fixed set of examples at a fixed moment, which is precisely what makes it comparable and precisely why it stops being true. Inputs drift. Customers start describing a new product with vocabulary that did not exist when the set was frozen, a competitor changes their pricing and the questions people ask change shape, and the system that scored ninety-four in March is scoring eighty-eight in September without anybody having touched a line of it.

So the evaluation needs a companion that runs continuously, and it measures something different: not whether the answers are right, which nobody can know in production without labels, but whether the inputs still look like the inputs the system was evaluated on.

• The distribution of request lengths.
• The rate of inputs the system declines to answer.
• The proportion falling into each category.

None of these need a correct answer to be useful, and all of them move before accuracy does.

The practical arrangement is to re-run the frozen evaluation on a schedule — quarterly is usually enough — and to watch the input distribution daily. The quarterly run tells you whether the system still works; the daily watch tells you when to run it early. Teams that do only the first find out about drift one quarter late, and teams that do only the second have a dashboard of moving lines with nothing to anchor them to.

The examples you get wrong are the asset

When an evaluation finishes, the most valuable output is not the score but the list of examples the system got wrong, and that list should be read by a person rather than counted. Thirty minutes with the failures almost always produces a classification that no metric would have suggested.

• A third of them turn out to be mislabelled in the set itself.
• A third share a single recognisable cause.
• The remainder are genuinely hard.

That first third matters more than it appears. Mislabelled examples put a hard ceiling on the measured score that has nothing to do with the system, and fixing them is the single cheapest improvement available in most projects — though it has to be done by someone who did not see which examples the system failed, or the correction becomes a quiet form of tuning the test to the answer.

The middle third is where the engineering is. A single recognisable cause — dates in an unusual format, a product name that collides with a common word, inputs arriving with the fields in a different order — is a fix rather than a research problem, and it is typically worth more points than changing the model. We have twice replaced a planned model upgrade with an afternoon of input normalisation that recovered more accuracy than the upgrade was projected to.

Keep the failures as a second set

Collect the genuinely hard failures into their own small file and run it alongside the main set. It is a deliberately pessimistic measurement and it moves much faster than the aggregate, which makes it a better signal when you are iterating. It also stops the most demoralising pattern in this work, where a change improves the overall figure by half a point and nobody can tell whether the thing they were actually trying to fix got fixed.

What the result does not tell you

Every evaluation report we write ends with a paragraph on its own limits, and it is the section clients quote back to us most often. A score on a frozen set does not tell you how the system behaves on inputs unlike anything in the set, how it fails when an upstream service is slow, what it does with a deliberately adversarial input, or whether the people who have to work alongside it will trust it. Those are all real questions and none of them are answered by the number.

Writing that paragraph is uncomfortable and it is the most useful thing in the document, because it converts an implied promise into a stated scope. A client who has read it does not later discover a gap and conclude they were oversold; they discover a gap they were told about and ask what it would cost to close. The difference between those two conversations is the difference between a second project and a dispute.

It also disciplines the evaluation itself. Once you have to write down what the measurement does not cover, you notice which of those gaps could have been covered cheaply, and in practice about half of them get added to the set before the report goes out. The paragraph is therefore not only a disclaimer; it is the last and most productive review step in the whole exercise.

What we put in an evaluation report

Ours are short and they always contain the same sections, because the value is in the comparability rather than the prose. The set and where it came from, with the date it was frozen. The three scores. The per-class breakdown. The cost table. The agreement ceiling. The spread across repeated runs. And a paragraph at the end stating what we think the result does not tell you, which is the section that has most often changed a client’s decision.

• The evaluation set, its origin and the date it was frozen.
• Three scores: the system, the majority class, the current process.
• A per-class breakdown with the worst class first.
• The cost of each mistake type, in business units.
• Human agreement on the same task, as the ceiling.
• The spread across three runs, for generative systems.

None of this makes an evaluation expensive. A first honest one is usually two or three days, most of which is spent assembling the set and arguing productively about what the right answer is. That argument is the valuable part. More than once the evaluation has ended the project early, because defining correctness precisely revealed that the task as described could not be done, and finding that out in week one is the cheapest possible outcome.