Nobody published the denominator

You sell a compiled program. The buyer owns the machine it runs on, so they can take it apart at leisure.

Obfuscation is the defence: a compiler pass that leaves the program’s behaviour alone while making its compiled form harder to read back. Every paper measuring how well it holds against AI code-reversers reports one number — how far the reverser’s success rate fell.

That number averages two populations that behave nothing alike. There are the functions the transform changed. And there are the functions it never touched, which recover exactly as well as they did before anyone tried to protect them, and get counted into the obfuscated result anyway.

So the headline can move because the defence worked, or because the transform covered less ground. Buyers choose on those published numbers, because those numbers are the only evidence anyone has.

Nothing that gets published lets a reader tell the two cases apart.

The defect is arithmetic before it is anything about binaries. This series measures it in neural decompilation because that is a setting where each missing term can still be put back and counted: whether the transform fired, whether the model read the function whole, which toolchain and which disassembler rendered the text, and what the attacker had already seen.

The one number everybody reports¶

The metric deserves a steelman, because the arithmetic behind it is sound.

A neural decompiler is a transformer that reads stripped assembly — machine instructions with the function and variable names thrown away — and emits compilable C. The obfuscation passes it is tested against are the ones above: flattening control flow into a dispatch loop, injecting branches that never execute, encrypting strings and constants so they materialize only at runtime. The attacker downloads the shipped binary and runs a decompiler over it, so the field’s measurement is the natural one: decompile clean, decompile obfuscated, report how far recovery fell.

Recovery is defined strictly. A function counts as recovered when the emitted C compiles and reproduces the reference behaviour on its test inputs — output that merely looks like the original scores zero. A second family of metrics scores the shape of what came back, usually the recovered control-flow graph’s similarity to the original. Different question, different axis; every number here stays on the re-executability axis, and any pair that needs a bridge between the two gets one in the text.

So: one scalar, strictly defined, cheap, comparable across papers, shaped exactly like the accuracy-drop-under-attack number adversarial machine learning has reported for a decade. It is a reasonable thing to publish. What goes unpublished is the population it was computed over.

What thirty-seven papers publish¶

We built a census to find out whether that omission was ours or the field’s. The corpus is 96 papers after deduplication, 96 papers, unrelated to the 96 configurations of our own grid further down. Of those, 91 were verified against their full text, and 37 report an obfuscation-strength number of the kind above.

We then scored each of those 37 for the four things a reader needs to recompute the number. Conditioning the rate on whether the transform actually fired on the function being scored, fired meaning the compiled object’s bytes for that function change relative to the clean build: 1 of 37. Stratifying it by truncation, these models read a fixed number of tokens, and anything longer is cut off and scored as a failure, which the fifth section takes apart, so a reader can tell how many instances the model never saw whole: 0 of 37. Decomposing the drop by toolchain provenance, meaning which compiler built the binaries, since two compilers turn identical source into visibly different assembly: 0 of 37. Reporting a bare aggregate with none of the above: 35 of 37.

The single positive in the firing-conditioned column carries a disqualifier, and the disqualifier travels with the count everywhere it is quoted. That paper, DisasLLM, named in its row of the scored table the deposit ships, does condition on firing, but its firing-conditioned cell is a disassembly precision/recall/F1 row rather than a re-executability recovery cell. On the axis this piece is about, the count is zero.

The finding is deliberately unexciting: nobody is cheating. Every one of those 35 numbers appears to be correctly computed over whatever population its authors used. There is simply no way for a reader to learn what that population was, no way to compare two papers’ numbers, no way to tell a strong transform from a lucky denominator. The deposit ships the scored table and the tally script, so the counts can be recomputed rather than trusted.

The functions the transform skipped¶

Here is the mechanism that makes the scalar un-auditable.

Firing is a per-function property. A transform that flattens control flow needs control flow to flatten; a pass that encrypts string literals needs string literals; a pass gated on a size heuristic skips whatever falls below the threshold. So on any real corpus some functions come out changed and some come out byte-identical to the clean build. Both are handed to the decompiler, and both are counted in one average:

aggregate recovery
  =  share of functions the transform fired on
       x  recovery among those functions
  +  share it never fired on
       x  recovery among those        <-- which is just the clean rate

The first term is the only one that answers the question anyone asked. The second is clean-rate contamination: functions the transform never modified, still recovering at their pre-obfuscation rate, arriving in the obfuscated column as though the transform had been tested on them. Push the firing rate down and the aggregate drifts toward the clean rate no matter how effective the transform is on the functions it does reach.

Put round numbers on it. A transform fires on 20 of 100 functions and recovers nothing at all on those 20, total defeat on everything it touched. The other 80 were never modified, so they recover at the clean rate of 10%, which is 8 functions. The published aggregate is 8%, down from 10%, and it reads as a transform that barely helped. The 20 it actually reached were annihilated.

The never-fired population is wildly heterogeneous. Some configurations in our grid fire on every function and have no ghosts at all; sparse, heuristic-gated ones fire on a small minority and are almost entirely ghosts. The distortion therefore varies per cell, no rule of thumb corrects for it afterwards, and only the person who ran the transform can publish it.

Conditioning on firing has a cost, and the cost is honest: the denominator shrinks and the confidence interval around the rate, the range the true value plausibly occupies, given how few functions were counted, widens with it. Zero successes out of two dozen functions and zero out of a hundred and fifty carry the same point estimate and very different amounts of information. The wide interval tells you the experiment was small, which the aggregate hid by padding the denominator.

The same fix, pointed at our own grid, cost us a result we had already written down.

A word on structure, because the rest leans on it: the grid is a 96-configuration factorial of our own obfuscation toggles, each run over the same 164 functions, 95 obfuscated configurations plus the clean baseline, which is itself a point inside the grid, so every grid-wide statistic below is computed over 95. A cell is one configuration’s number together with the population it was computed over; an arm or leg is a whole sub-experiment with its own model, scorer and window.

Every cell in this section comes from one arm: the stock released LLM4Decompile v1.5 over 164 functions per configuration, scored with the strict clang-side re-executability scorer at an 8192-token context window, the amount of disassembly the model can read at once, on x86 binaries built at O2, the setting most shipped software uses, seed 0xC0FFEE. Read the raw aggregates and a clean story is sitting right there: control-flow flattening leaves recovery at 6.1% while bogus control flow crushes it to 0.6%. Ten to one. Injecting dead branches looks dramatically stronger than restructuring control flow, which is counterintuitive, publishable, and exactly the kind of result that makes a grid worth running. We wrote it into the table as a ranking.

Conditioning both cells on firing collapses them into each other. Both land in the 0–1.4% band, and Fisher’s exact test, the right test when counts are small enough that a chi-squared approximation misbehaves, separates nothing.

Which number you quote depends on the denominator you tested, so both belong in print. Across the full-164 cells, each transform’s fired-only successes carried over the whole corpus, every pairwise contrast returns p > 0.3, with a minimum of 0.50. Across the fired-only cells, where each transform carries its own denominator, the minimum falls to p = 0.22: control-flow flattening’s 2 of 139 against constant encryption’s 0 of 157, the third transform in the same collapsed band and the pair that comes nearest to separating anything. Attaching the roomier full-N figure to a fired-only cell is precisely the splice this piece exists to name, so it does not happen here. Neither figure buys an ordering. At two successes between the two cells, 0.22 is what the arithmetic returns when there is nothing there to find.

The dissolution runs in the direction that costs the most, and the arithmetic is small enough to check by hand. Bogus control flow fires on all 164 functions, so its 0.6% was already a fired-only number and does not move. Control-flow flattening fires on 139 and skips 25, and 8 of its 10 apparent successes sit in those 25, functions the pass never touched, recovering at 32% against this arm’s 7.3% clean rate, because they were the easy ones it declined. Scored over the 139 it reached, flattening reads 1.4% (2/139) against bogus control flow’s 0.6% (1/164).

The collapse licenses one sentence and no more. It does not establish that the two transforms are equivalent: at the success counts these cells contain, the data pays for exactly no ordering established at this n, and an equivalence claim would need a powered equivalence test this cell cannot support.

We killed the ranking. A protocol that only ever indicts other people’s numbers is a marketing document.

A no-op that passes¶

The ranking dissolution is a corollary. This is the wall.

Somewhere in the grid sits a backend-marker pass: it writes a short tag into the object file recording which code generator produced it, a build stamp, and changes nothing about the program’s structure, its control flow, or its data. Its per-function results are worth stating exactly, because the exactness is the entire argument.

They come from a different checkpoint than the ranking above, the fine-tuned clang-native 6.7B control attacker rather than the stock release, on the same strict clang-side scorer, the same 8192-token window, the same 164 functions at x86/O2, the same seed. Two checkpoints, so the two sections’ rates cannot be compared with each other.

It fires on 6 of the 164 functions: six compiled objects whose bytes differ from the clean build. The other 158 are byte-identical, every byte, and you can put the two hex dumps side by side and look. Its fired-only recovery is 0 of 6. And its per-function outcome triples, recovered, compiled, truncated, are identical to the clean build on all 164 functions, with 0 differences and the same 15 recovered functions. Its aggregate recovery is therefore unchanged from that arm’s clean rate of 9.15%.

Two things about that 9.15% before anyone quotes it. It is low because of the scorer: score the same checkpoint on the same 164 clean functions with the permissive gcc-side scorer and the clean rate reads 15.9% (26/164), same model, same window, same functions, one scoring rule apart, which is why no clean rate in this series travels without the scorer that produced it. And the pass is aggregate-identical under the stock checkpoint too, the same 12 functions recovering with and without it, so the blindness below survives a change of attacker.

The field’s standard metric reports no meaningful drop here, which is correct, and which is the problem: it had no way to distinguish “this transform does nothing” from “this transform does nothing detectable by this attacker on this corpus” from “this transform was never applied to most of the corpus.” All three read as the same scalar. A metric that cannot fail a no-op cannot certify a transform.

What the configuration demonstrates is that blindness, and only that. All six functions it perturbs were already unrecoverable in the clean build, so the cell has close to no statistical power to detect an effect if one existed. It cannot serve as the protocol’s load-bearing negative control; sold as a powered null it would invert under the first competent reviewer.

The wall the fix does not reach¶

Conditioning on firing fixes coverage. It does nothing whatsoever about the second confound, and the second confound is arguably worse because it is invisible from inside the metric.

Neural decompilers have a context window. Feed one a disassembly listing longer than that window and the input is cut off; the model answers on a fragment, fails to produce compilable output, and the harness scores the instance as a failure. Which it is, a failure of the measurement apparatus, telling you only that the listing was long. Obfuscation passes that duplicate blocks or inject dispatch loops make listings much longer, so they push instances over the window and harvest failures the model never had a chance at. What moved was the truncation rate.

The scale is larger than it sounds. In the o-mvll arm of our own grid, the same fine-tuned control attacker, the same strict clang-side scorer and the same 8192-token window as above, reading llvm-objdump’s rendering of the disassembly, exactly 1 of 164 functions came back working. As an aggregate that is roughly a 99% failure rate, and the field’s vocabulary would call it near-total deterrence. Stratified, it is 81.7% truncation (134 of 164) plus a fired-only recovery of 3.3% (1 of 30) on the functions the model saw whole, whose 95% interval of [0.6, 16.7] comfortably contains that arm’s own 9.15% clean rate.

And a truncation rate is not a property of the transform. It is a property of the (window, disassembler, tokenizer) triple: the same configuration, disassembled by a different tool or tokenized by a different model, truncates at a different rate, and at a smaller window it truncates more. So a truncation rate cannot be carried across arms or rescaled between windows, it has to be recomputed inside each arm and printed on each cell, in the same breath as the recovery number it explains. Zero of the 37 papers do this. Neither did we, in the first cut of our own grid.

The two confounds compound in a way worth naming. Coverage dilution pulls the aggregate up toward the clean rate by padding it with untested functions; truncation pushes it down by scoring unseen functions as failures. Both are population effects, both are invisible in the scalar, and they do not cancel in any predictable ratio.

We can put a ratio on the dilution, what fraction of a configuration’s aggregate recovery comes from functions the transform never fired on, and across the 95 obfuscated configurations it is bimodal, two piles at the ends of the axis. 48 score zero: they fire on all 164 functions, so there is no ghost population to dilute anything, and that bucket absorbs the configurations where nothing recovered at all and the ratio is a 0/0 the code prints as zero. 43 score close to 1.0, meaning almost the entire aggregate is clean-rate contamination. Four sit mid-band near 0.47, and they are the cells whose only transform rewrites arithmetic into mixed boolean-arithmetic expressions, the one corner of the grid where the never-fired population is neither absent nor overwhelming. So the scalar diagnoses whether never-fired functions exist, and against an attacker crushed to near zero on the functions it does reach, that is the whole of its range.

Which leaves the denominators themselves, the mundane failure mode that eats the most numbers. Four of them run through this series:

• 164, the grid corpus: every function, in every configuration.
• 149, the matched non-truncated subset: functions whose disassembly fits a 4096-token window on both sides of the next article’s compiler comparison. Fifteen truncate and are dropped from both arms, a different fifteen from the fifteen the placebo section recovers, sharing exactly one member. There is one 149, not two: the two legs that use it were checked function by function and carry the same set.
• 151, the fired, non-truncated population of one obfuscated cell in the third article’s scaling ladder, at that arm’s own window and prompt. It sits inside 164 and inside 162, but 7 of its functions fall outside the 149: the two overlap without either containing the other.
• 162, the grid minus the two functions that truncate in the disassembler-format comparison.

None is a superset the obfuscated cells are drawn from; each obfuscated cell has its own fired, non-truncated population. Put flattening’s 2/139 beside bogus control flow’s 1/164 and both rates are honest; put either beside the 10/164 aggregate that produced the ranking and you have drawn a trend across three different populations.

What this does not show¶

The limits are specific, and three of them are load-bearing.

No published number here is disputed. The census makes no claim that any of the 35 bare aggregates is arithmetically wrong. What it claims is that the presentation is un-auditable: the population, the firing rate and the truncation rate are missing, so a reader cannot recompute the cell or compare it with another paper’s. That is a reporting finding, not an allegation of arithmetic error.

The census’s own reliability is limited. Single-pass scoring, adversarially re-verified on 70 of 96 papers, 233 of 235 re-examined cells upheld, and no computable inter-annotator statistic. That is stronger than one pass alone and weaker than validated dual annotation, and the paper says so in those words.

Every attacker result above is scoped, and the scope is narrow. These are static, client-side measurements against two 6.7B decompiler checkpoints on 164 functions, one scorer and one window at a time, single-shot: no tool use, no iteration, no human in the loop. Every number here is a floor an attacker starts from. A better-resourced attacker is the subject of the third piece in this series, and it moves them substantially. Nothing here says any shipping product is broken. Client-side hardening is deterrence, never a guarantee.

Two smaller limits. The backend-marker configuration’s null is unpowered, all six of its firings landed on functions that were already unrecoverable when clean, so it evidences the metric’s blindness and nothing about the transform. And the first minted version of the deposit calls that configuration “byte-inert,” one page from the definition of firing as a byte difference; the pass changes the bytes of six functions and leaves the outcomes alone, so the correct word is outcome-inert, and the correction goes into the next version.

One bridge, rather than a limit. Both arms here run the strict clang-side scorer at an 8192-token window; the legs in the next two articles run a more permissive gcc-side scorer at the model’s native 4096-token window, which is a different measurement and cannot go on the same axis without saying so out loud. It is also why the two transforms above trade places between articles: at these counts the ordering is a property of the measurement regime.

What an obfuscation number has to carry¶

| what read fine | what was actually true | the rule it became | |, |, |, | | a tenfold gap between two transforms is a ranking | 8 of the wider cell’s 10 successes were on functions the pass never fired on; conditioned, the two land at 1.4% and 0.6%, and Fisher’s minimum is p = 0.22 across the fired-only cells (0.50 across the full-164 ones) | report the rate over the functions the transform actually reached, or report no ordering | | the aggregate fell, so the transform works | the aggregate’s second term is the clean rate of functions the transform never fired on | publish the firing rate per cell, or the drop is unattributable | | near-total deterrence in the o-mvll cell | 81.7% of that cell’s functions exceeded the context window and were scored as failures unseen; 1 of the 30 seen whole came back | print the (window, disassembler, tokenizer) triple and the truncation rate on every cell, recomputed per arm | | a no-op will obviously fail the metric | outcomes identical to clean on all 164 functions, aggregate unchanged from that arm’s 9.15%, and the metric passed it | require a metric a no-op fails, or it certifies nothing | | our negative control settles the point | all six of its firings landed on functions that were already unrecoverable when clean | a null needs power; state when yours has almost none |

Thirty-five papers reported a drop and did not say what it was a drop over. Our own grid reported a ten-to-one gap between two transforms and did not say that four fifths of the wider cell’s successes came from functions the pass had declined to touch. Same defect, wildly different consequence, and both of them invisible to any reader who only has the number.

The next article changes the scope from the binary to the environment it was built in, two arms, the same functions, zero obfuscation in either one, and a gap nobody put there on purpose.

The census is one subfield; the rule it leaves behind is not. Any eval that folds inputs a transform never touched into the same average, without reporting how often it fired, inherits this exact bug.