Can someone independently recompute a published benchmark number and tell me whether it holds?

Often, and finding out which is most of the work. A recompute takes one number, re-runs it from the published artifacts alone, and returns one of three verdicts: it reproduces, it reproduces at a different value, or it cannot be recomputed and here is the artifact that was missing.

The scope, stated narrowly on purpose¶

A recompute is not a review of your evaluation programme, and it is not an opinion about whether the method was a good idea. It takes one number — the one in the abstract, the one on the slide, the one a purchase is about to be made on — and asks whether a stranger working only from what was published arrives at the same figure.

That is a smaller question than “is this research any good”, and it is the one with an answer. It is also the question everybody skips, because it looks like it should be trivial and is not.

The five things that stop a recompute, in roughly the order they bite¶

Nothing fixed the randomness. No seed in the paper, no seed in the repo, or a seed that controls the shuffling but not the sampler. This is worse now than it was: a model served through a gateway may not expose a seed at all, so the run is unrepeatable by construction and no amount of care downstream recovers it. We hit exactly that — the apparatus behind our own 50-run measurement notes the sampler was not seedable through the gateway, so the honest move was to report the spread rather than a seed nobody could set.

Nothing pinned the environment. A version range in requirements.txt is not a pin, and the compiler, the OS image and the driver are part of the environment even when nobody thinks of them as dependencies. This is not a pedantic point. In our own controlled grid, the same 149 matched functions, the same released model and one scorer for both arms — with no obfuscation applied in either — recovered at 27.5% from GCC-built binaries and 15.4% from clang-built ones, a paired McNemar p of 0.0029. A published number can carry a double-digit term from a build choice it never names.

Nobody stated the denominator. A rate is a count over a population, and the population is the part that goes unreported. Across 37 papers reporting an obfuscation-strength number, 1 conditions that number on whether the transform actually fired on the function being scored, and 0 stratify it by context-window truncation. When the population is unstated, there is no recompute to attempt: there is only a guess at what set the author had in mind.

The evaluation code is not the code in the paper. The published repository is usually a later, tidier thing than the one that produced the table. Off-by-one in a filter, a case-insensitive comparison that used to be case sensitive, an exclusion added during cleanup — each moves the headline without moving the method section. Recomputing is how you find out; reading is not.

Scoring changed underneath the number. Strict versus lenient matching, the token window the model was given, the checkpoint it was served from. Each is a free parameter, each is frequently unreported, and each is enough to move a result on its own. Fine-tuning the attacker in our grid recovered 12–16% from binaries that had not changed at all.

The three verdicts, and why the third one is a result¶

A recompute ends in one of three places.

It reproduces. The figure comes back within a stated tolerance, and you now have a re-runnable procedure that produces it. This is the outcome people expect and the least common of the three.

It reproduces at a different value. The procedure runs to completion and lands somewhere else. Now the interesting work starts, because the gap is attributable: a version, a filter, a scorer setting, an excluded set.

It cannot be recomputed. Some artifact needed to run the procedure was never published and cannot be reconstructed. This is not a failed engagement. It is a finding about the number, and it is the finding that should change a buying decision, because a figure nobody outside the room can arrive at is a claim about a result rather than a measurement of one.

What none of those three is: a pass. A check that could not run has not passed, and any report that quietly rolls “we could not test this” into a green summary has destroyed the only thing it was hired to produce.

The instrument gets checked before the number does¶

An audit that only ever reports agreement is indistinguishable from an audit that cannot detect disagreement. So the harness is calibrated first, on inputs whose answer is already known — a negative control that should produce no effect, and a positive one that should produce a large one.

This matters more than it sounds. In our own grid, a control configuration whose per-function outcomes were identical to the clean build passed the field’s standard obfuscation-strength metric as a working obfuscator. The metric could not tell “did nothing” apart from “worked”. Had we run only the treated arms we would have reported a strength number and believed it.

The same discipline covers noise. One unchanged eval suite, one model, 50 runs at temperature zero: 8 runs scored 1.000 and 42 scored 0.947. Against that floor, a recomputed value two points from the published one is not a discrepancy, and reporting it as one would be its own kind of error.

What actually lands on the table¶

Three things, and it is worth checking a prospective auditor commits to all three.

A verdict on the named number, with the tolerance stated, so it is not an adjective. A procedure someone else can re-run — pinned environment, fixed seeds where seeds exist, the exact commands, and the intermediate outputs, so the audit itself is reproducible and you are not asked to trust it on the same terms you were asked to trust the original. And an explicit list of what could not be checked, with the reason for each, because that list is where the risk actually lives after the engagement ends.

When it is not worth doing¶

If the number is directional, if nothing is being decided on it, or if the authors already publish per-item outputs and a pinned container — in that last case, run it yourself; the work has been done for you and it is an afternoon.

The case for a recompute is narrow and it is specific: one figure, load-bearing for a decision, published by someone with an interest in it, in a form that nobody has yet re-run.