evaluations
An agent reads a sentence about your tool and decides, on that sentence, to call it. Your tests read the function. What a passing suite is evidence of, and what it is not.
evaluations
Read the whole series as one page- 01
- 02
Every verdict held. The score moved anyway.
An integration test has an oracle. An eval has a metric — and the metric is an accounting procedure nobody reviews. Four graders from four vendors agreed on every substantive verdict I gave them, three times each, and the pass rate still moved. Here is where an eval number actually comes from, why agents break it worse than models do, and the three things I now build before I report one.