What should an agent eval test that my unit tests never touch?

The gap, in one sentence¶

An agent decides to call your tool on the strength of a sentence about it. Your tests call the function.

Everything between those two facts is untested. The name, the one-line description, the parameter names, the type hints as they are rendered into the schema, the ordering, the examples, the error strings the model reads back after a failed call — all of that is input to the model’s decision and none of it is input to pytest. A suite at 100 percent coverage tells you the function works when called. It says nothing about whether it gets called, when it should, with what.

The five things worth testing that a unit suite structurally cannot¶

Selection. Given a request and the whole tool set, does the model pick the right tool? This is the test that breaks when you add the sixth tool and its description overlaps the second one. It is a property of the set, not of any member, which is why no per-function test can catch it.

Non-selection. Given a request that none of your tools serve, does the model decline rather than force the closest match? Agent suites overwhelmingly test the happy path and then ship a system that confidently calls a search tool when asked to delete something.

Argument construction. The model has to turn prose into your parameters. Ambiguous parameter names, an undocumented unit, an enum whose values are not in the schema — each produces a syntactically valid call with the wrong content, which your function will happily execute.

Recovery. When a call fails, the model reads your error message and decides what to do next. That message is a prompt. If it says ValueError: invalid input, you have told the model nothing and it will retry identically. Error text is part of the interface and belongs in the eval.

Trajectory. Multi-step tasks fail in the joins. Every individual step can be correct while the sequence loops, re-fetches, or stops one step early. This needs an eval over the whole run, and there is no unit-test analogue of it at all.

The measurement almost nobody makes first¶

Before you tune anything, measure your own harness’s variance. Run the unchanged suite against the unchanged model, at temperature zero, several dozen times, and look at the spread.

We did this: one eval suite, one model, 50 runs at temperature zero. Eight runs scored a perfect 1.000 and the remaining 42 scored 0.947. The suite did not change. The model did not change. The setting that is supposed to make output deterministic was on.

That spread is your noise floor, and it is the number that decides which of your future results mean anything. A 2-point improvement measured once, in a harness with that much run-to-run movement, is not an improvement. It is a sample. Most eval reporting we see does not include this measurement, which means most eval reporting cannot distinguish a real gain from a re-roll.

Where to start¶

Take the five categories above and write one case in each — five cases, not fifty. Then run your suite thirty times unchanged and write down the spread. You will learn more from that afternoon than from a month of adding cases, because everything you add afterwards is measured against a floor you actually know.