How do I check whether a published benchmark number measures what it claims to?

The one question that does most of the work¶

Almost every benchmark headline is a rate: a success count over a population. The count is usually reported. The population usually is not, and that is where the number goes wrong.

So the first question to ask of any published figure is: what was in the denominator, and what got left out of it? You are not accusing anyone of fabrication. You are asking for the thing that makes a rate a measurement rather than a summary statistic of an unspecified set.

The four ways a population quietly stops matching the claim¶

Items the treatment never touched. If the benchmark measures the effect of some transform — a defence, an optimisation, a mitigation — and the transform does not fire on every item, then the untouched items carry their untreated score straight into the total. The headline can move because the treatment worked or because it covered less ground, and a number that does not report firing rate cannot distinguish those two.

Items the system could not read whole. Anything longer than the model’s context window gets truncated and scored as a failure. That is a reading limit being reported as a property of the item. It inflates whatever the benchmark calls difficulty, and it does so unevenly, because item length is rarely independent of item type.

Items built by different tooling on the two sides. If the treated and untreated populations were produced by different compilers, different pipelines, or different versions, part of the measured gap is a dialect difference the system never trained on rather than the effect under study.

Items the system had already seen. Prior exposure changes the result, so a score is conditional on what the evaluated system had encountered before. Two runs that differ only in that condition are not comparable, and the condition is almost never stated.

The checklist¶

For a number you are about to buy on, or cite:

1. Is the population enumerated, or at least counted? Not “a benchmark of hard cases” — a number.
2. Is the exclusion rule stated, and were the excluded items counted?
3. If a transform is involved, is its firing rate reported per item?
4. Are truncations and timeouts distinguished from genuine failures?
5. Were both sides built and rendered by the same tooling?
6. Is per-item output published, or only the aggregate?
7. Could a stranger recompute the headline from what was published? If not, the number is a claim rather than a measurement.

Item 7 subsumes the rest and is the one worth memorising. A benchmark result is reproducible in the sense that matters when someone outside the room can arrive at the same figure from the published artifacts.

What the field actually reports¶

We ran the census rather than assuming. Across a 96-paper census of obfuscation-strength reporting — 91 verified against full text — 37 papers reported a strength number at all, and 35 of those reported it as a single bare aggregate. That is the base rate for the practice this checklist is asking after, in one subfield, measured.

Paired with that census is a 164-function grid of our own, run with a placebo negative control: a transform that should do nothing. It fired on 6 objects and changed zero per-function outcomes. That is what a negative control is for — it tells you the measurement apparatus is not generating signal on its own, and almost no published benchmark includes one.

The general shape¶

None of this is specific to security benchmarks or to code models. Any evaluation that folds items a treatment never touched into the same average, without reporting how often it fired, has the same bug. If you are reading a number and you cannot answer question 1, you do not yet know what the number measures — and neither, usually, does the person who published it.