measurement-validity

You sell a compiled program. The buyer owns the machine it runs on, so they can take it apart at leisure.

Obfuscation is the defence: a compiler pass that leaves the program’s behaviour alone while making its compiled form harder to read back. Every paper measuring how well it holds against AI code-reversers reports one number — how far the reverser’s success rate fell.

That number averages two populations that behave nothing alike. There are the functions the transform changed. And there are the functions it never touched, which recover exactly as well as they did before anyone tried to protect them, and get counted into the obfuscated result anyway.

So the headline can move because the defence worked, or because the transform covered less ground. Buyers choose on those published numbers, because those numbers are the only evidence anyone has.

Nothing that gets published lets a reader tell the two cases apart.

The defect is arithmetic before it is anything about binaries. This series measures it in neural decompilation because that is a setting where each missing term can still be put back and counted: whether the transform fired, whether the model read the function whole, which toolchain and which disassembler rendered the text, and what the attacker had already seen.

The one number everybody reports¶

The metric deserves a steelman, because the arithmetic behind it is sound.

A neural decompiler is a transformer that reads stripped assembly — machine instructions with the function and variable names thrown away — and emits compilable C. The obfuscation passes it is tested against are the ones above: flattening control flow into a dispatch loop, injecting branches that never execute, encrypting strings and constants so they materialize only at runtime. The attacker downloads the shipped binary and runs a decompiler over it, so the field’s measurement is the natural one: decompile clean, decompile obfuscated, report how far recovery fell.

Recovery is defined strictly. A function counts as recovered when the emitted C compiles and reproduces the reference behaviour on its test inputs — output that merely looks like the original scores zero. A second family of metrics scores the shape of what came back, usually the recovered control-flow graph’s similarity to the original. Different question, different axis; every number here stays on the re-executability axis, and any pair that needs a bridge between the two gets one in the text.

So: one scalar, strictly defined, cheap, comparable across papers, shaped exactly like the accuracy-drop-under-attack number adversarial machine learning has reported for a decade. It is a reasonable thing to publish. What goes unpublished is the population it was computed over.

What thirty-seven papers publish¶

We built a census to find out whether that omission was ours or the field’s. The corpus is 96 papers after deduplication, 96 papers, unrelated to the 96 configurations of our own grid further down. Of those, 91 were verified against their full text, and 37 report an obfuscation-strength number of the kind above.

We then scored each of those 37 for the four things a reader needs to recompute the number. Conditioning the rate on whether the transform actually fired on the function being scored, fired meaning the compiled object’s bytes for that function change relative to the clean build: 1 of 37. Stratifying it by truncation, these models read a fixed number of tokens, and anything longer is cut off and scored as a failure, which the fifth section takes apart, so a reader can tell how many instances the model never saw whole: 0 of 37. Decomposing the drop by toolchain provenance, meaning which compiler built the binaries, since two compilers turn identical source into visibly different assembly: 0 of 37. Reporting a bare aggregate with none of the above: 35 of 37.

The single positive in the firing-conditioned column carries a disqualifier, and the disqualifier travels with the count everywhere it is quoted. That paper, DisasLLM, named in its row of the scored table the deposit ships, does condition on firing, but its firing-conditioned cell is a disassembly precision/recall/F1 row rather than a re-executability recovery cell. On the axis this piece is about, the count is zero.

The finding is deliberately unexciting: nobody is cheating. Every one of those 35 numbers appears to be correctly computed over whatever population its authors used. There is simply no way for a reader to learn what that population was, no way to compare two papers’ numbers, no way to tell a strong transform from a lucky denominator. The deposit ships the scored table and the tally script, so the counts can be recomputed rather than trusted.

The functions the transform skipped¶

Here is the mechanism that makes the scalar un-auditable.

Firing is a per-function property. A transform that flattens control flow needs control flow to flatten; a pass that encrypts string literals needs string literals; a pass gated on a size heuristic skips whatever falls below the threshold. So on any real corpus some functions come out changed and some come out byte-identical to the clean build. Both are handed to the decompiler, and both are counted in one average:

aggregate recovery
  =  share of functions the transform fired on
       x  recovery among those functions
  +  share it never fired on
       x  recovery among those        <-- which is just the clean rate

The first term is the only one that answers the question anyone asked. The second is clean-rate contamination: functions the transform never modified, still recovering at their pre-obfuscation rate, arriving in the obfuscated column as though the transform had been tested on them. Push the firing rate down and the aggregate drifts toward the clean rate no matter how effective the transform is on the functions it does reach.

Put round numbers on it. A transform fires on 20 of 100 functions and recovers nothing at all on those 20, total defeat on everything it touched. The other 80 were never modified, so they recover at the clean rate of 10%, which is 8 functions. The published aggregate is 8%, down from 10%, and it reads as a transform that barely helped. The 20 it actually reached were annihilated.

The never-fired population is wildly heterogeneous. Some configurations in our grid fire on every function and have no ghosts at all; sparse, heuristic-gated ones fire on a small minority and are almost entirely ghosts. The distortion therefore varies per cell, no rule of thumb corrects for it afterwards, and only the person who ran the transform can publish it.

Conditioning on firing has a cost, and the cost is honest: the denominator shrinks and the confidence interval around the rate, the range the true value plausibly occupies, given how few functions were counted, widens with it. Zero successes out of two dozen functions and zero out of a hundred and fifty carry the same point estimate and very different amounts of information. The wide interval tells you the experiment was small, which the aggregate hid by padding the denominator.

The same fix, pointed at our own grid, cost us a result we had already written down.

A word on structure, because the rest leans on it: the grid is a 96-configuration factorial of our own obfuscation toggles, each run over the same 164 functions, 95 obfuscated configurations plus the clean baseline, which is itself a point inside the grid, so every grid-wide statistic below is computed over 95. A cell is one configuration’s number together with the population it was computed over; an arm or leg is a whole sub-experiment with its own model, scorer and window.

Every cell in this section comes from one arm: the stock released LLM4Decompile v1.5 over 164 functions per configuration, scored with the strict clang-side re-executability scorer at an 8192-token context window, the amount of disassembly the model can read at once, on x86 binaries built at O2, the setting most shipped software uses, seed 0xC0FFEE. Read the raw aggregates and a clean story is sitting right there: control-flow flattening leaves recovery at 6.1% while bogus control flow crushes it to 0.6%. Ten to one. Injecting dead branches looks dramatically stronger than restructuring control flow, which is counterintuitive, publishable, and exactly the kind of result that makes a grid worth running. We wrote it into the table as a ranking.

Conditioning both cells on firing collapses them into each other. Both land in the 0–1.4% band, and Fisher’s exact test, the right test when counts are small enough that a chi-squared approximation misbehaves, separates nothing.

Which number you quote depends on the denominator you tested, so both belong in print. Across the full-164 cells, each transform’s fired-only successes carried over the whole corpus, every pairwise contrast returns p > 0.3, with a minimum of 0.50. Across the fired-only cells, where each transform carries its own denominator, the minimum falls to p = 0.22: control-flow flattening’s 2 of 139 against constant encryption’s 0 of 157, the third transform in the same collapsed band and the pair that comes nearest to separating anything. Attaching the roomier full-N figure to a fired-only cell is precisely the splice this piece exists to name, so it does not happen here. Neither figure buys an ordering. At two successes between the two cells, 0.22 is what the arithmetic returns when there is nothing there to find.

The dissolution runs in the direction that costs the most, and the arithmetic is small enough to check by hand. Bogus control flow fires on all 164 functions, so its 0.6% was already a fired-only number and does not move. Control-flow flattening fires on 139 and skips 25, and 8 of its 10 apparent successes sit in those 25, functions the pass never touched, recovering at 32% against this arm’s 7.3% clean rate, because they were the easy ones it declined. Scored over the 139 it reached, flattening reads 1.4% (2/139) against bogus control flow’s 0.6% (1/164).

The collapse licenses one sentence and no more. It does not establish that the two transforms are equivalent: at the success counts these cells contain, the data pays for exactly no ordering established at this n, and an equivalence claim would need a powered equivalence test this cell cannot support.

We killed the ranking. A protocol that only ever indicts other people’s numbers is a marketing document.

A no-op that passes¶

The ranking dissolution is a corollary. This is the wall.

Somewhere in the grid sits a backend-marker pass: it writes a short tag into the object file recording which code generator produced it, a build stamp, and changes nothing about the program’s structure, its control flow, or its data. Its per-function results are worth stating exactly, because the exactness is the entire argument.

They come from a different checkpoint than the ranking above, the fine-tuned clang-native 6.7B control attacker rather than the stock release, on the same strict clang-side scorer, the same 8192-token window, the same 164 functions at x86/O2, the same seed. Two checkpoints, so the two sections’ rates cannot be compared with each other.

It fires on 6 of the 164 functions: six compiled objects whose bytes differ from the clean build. The other 158 are byte-identical, every byte, and you can put the two hex dumps side by side and look. Its fired-only recovery is 0 of 6. And its per-function outcome triples, recovered, compiled, truncated, are identical to the clean build on all 164 functions, with 0 differences and the same 15 recovered functions. Its aggregate recovery is therefore unchanged from that arm’s clean rate of 9.15%.

Two things about that 9.15% before anyone quotes it. It is low because of the scorer: score the same checkpoint on the same 164 clean functions with the permissive gcc-side scorer and the clean rate reads 15.9% (26/164), same model, same window, same functions, one scoring rule apart, which is why no clean rate in this series travels without the scorer that produced it. And the pass is aggregate-identical under the stock checkpoint too, the same 12 functions recovering with and without it, so the blindness below survives a change of attacker.

The field’s standard metric reports no meaningful drop here, which is correct, and which is the problem: it had no way to distinguish “this transform does nothing” from “this transform does nothing detectable by this attacker on this corpus” from “this transform was never applied to most of the corpus.” All three read as the same scalar. A metric that cannot fail a no-op cannot certify a transform.

What the configuration demonstrates is that blindness, and only that. All six functions it perturbs were already unrecoverable in the clean build, so the cell has close to no statistical power to detect an effect if one existed. It cannot serve as the protocol’s load-bearing negative control; sold as a powered null it would invert under the first competent reviewer.

The wall the fix does not reach¶

Conditioning on firing fixes coverage. It does nothing whatsoever about the second confound, and the second confound is arguably worse because it is invisible from inside the metric.

Neural decompilers have a context window. Feed one a disassembly listing longer than that window and the input is cut off; the model answers on a fragment, fails to produce compilable output, and the harness scores the instance as a failure. Which it is, a failure of the measurement apparatus, telling you only that the listing was long. Obfuscation passes that duplicate blocks or inject dispatch loops make listings much longer, so they push instances over the window and harvest failures the model never had a chance at. What moved was the truncation rate.

The scale is larger than it sounds. In the o-mvll arm of our own grid, the same fine-tuned control attacker, the same strict clang-side scorer and the same 8192-token window as above, reading llvm-objdump’s rendering of the disassembly, exactly 1 of 164 functions came back working. As an aggregate that is roughly a 99% failure rate, and the field’s vocabulary would call it near-total deterrence. Stratified, it is 81.7% truncation (134 of 164) plus a fired-only recovery of 3.3% (1 of 30) on the functions the model saw whole, whose 95% interval of [0.6, 16.7] comfortably contains that arm’s own 9.15% clean rate.

And a truncation rate is not a property of the transform. It is a property of the (window, disassembler, tokenizer) triple: the same configuration, disassembled by a different tool or tokenized by a different model, truncates at a different rate, and at a smaller window it truncates more. So a truncation rate cannot be carried across arms or rescaled between windows, it has to be recomputed inside each arm and printed on each cell, in the same breath as the recovery number it explains. Zero of the 37 papers do this. Neither did we, in the first cut of our own grid.

The two confounds compound in a way worth naming. Coverage dilution pulls the aggregate up toward the clean rate by padding it with untested functions; truncation pushes it down by scoring unseen functions as failures. Both are population effects, both are invisible in the scalar, and they do not cancel in any predictable ratio.

We can put a ratio on the dilution, what fraction of a configuration’s aggregate recovery comes from functions the transform never fired on, and across the 95 obfuscated configurations it is bimodal, two piles at the ends of the axis. 48 score zero: they fire on all 164 functions, so there is no ghost population to dilute anything, and that bucket absorbs the configurations where nothing recovered at all and the ratio is a 0/0 the code prints as zero. 43 score close to 1.0, meaning almost the entire aggregate is clean-rate contamination. Four sit mid-band near 0.47, and they are the cells whose only transform rewrites arithmetic into mixed boolean-arithmetic expressions, the one corner of the grid where the never-fired population is neither absent nor overwhelming. So the scalar diagnoses whether never-fired functions exist, and against an attacker crushed to near zero on the functions it does reach, that is the whole of its range.

Which leaves the denominators themselves, the mundane failure mode that eats the most numbers. Four of them run through this series:

• 164, the grid corpus: every function, in every configuration.
• 149, the matched non-truncated subset: functions whose disassembly fits a 4096-token window on both sides of the next article’s compiler comparison. Fifteen truncate and are dropped from both arms, a different fifteen from the fifteen the placebo section recovers, sharing exactly one member. There is one 149, not two: the two legs that use it were checked function by function and carry the same set.
• 151, the fired, non-truncated population of one obfuscated cell in the third article’s scaling ladder, at that arm’s own window and prompt. It sits inside 164 and inside 162, but 7 of its functions fall outside the 149: the two overlap without either containing the other.
• 162, the grid minus the two functions that truncate in the disassembler-format comparison.

None is a superset the obfuscated cells are drawn from; each obfuscated cell has its own fired, non-truncated population. Put flattening’s 2/139 beside bogus control flow’s 1/164 and both rates are honest; put either beside the 10/164 aggregate that produced the ranking and you have drawn a trend across three different populations.

What this does not show¶

The limits are specific, and three of them are load-bearing.

No published number here is disputed. The census makes no claim that any of the 35 bare aggregates is arithmetically wrong. What it claims is that the presentation is un-auditable: the population, the firing rate and the truncation rate are missing, so a reader cannot recompute the cell or compare it with another paper’s. That is a reporting finding, not an allegation of arithmetic error.

The census’s own reliability is limited. Single-pass scoring, adversarially re-verified on 70 of 96 papers, 233 of 235 re-examined cells upheld, and no computable inter-annotator statistic. That is stronger than one pass alone and weaker than validated dual annotation, and the paper says so in those words.

Every attacker result above is scoped, and the scope is narrow. These are static, client-side measurements against two 6.7B decompiler checkpoints on 164 functions, one scorer and one window at a time, single-shot: no tool use, no iteration, no human in the loop. Every number here is a floor an attacker starts from. A better-resourced attacker is the subject of the third piece in this series, and it moves them substantially. Nothing here says any shipping product is broken. Client-side hardening is deterrence, never a guarantee.

Two smaller limits. The backend-marker configuration’s null is unpowered, all six of its firings landed on functions that were already unrecoverable when clean, so it evidences the metric’s blindness and nothing about the transform. And the first minted version of the deposit calls that configuration “byte-inert,” one page from the definition of firing as a byte difference; the pass changes the bytes of six functions and leaves the outcomes alone, so the correct word is outcome-inert, and the correction goes into the next version.

One bridge, rather than a limit. Both arms here run the strict clang-side scorer at an 8192-token window; the legs in the next two articles run a more permissive gcc-side scorer at the model’s native 4096-token window, which is a different measurement and cannot go on the same axis without saying so out loud. It is also why the two transforms above trade places between articles: at these counts the ordering is a property of the measurement regime.

What an obfuscation number has to carry¶

| what read fine | what was actually true | the rule it became | |, |, |, | | a tenfold gap between two transforms is a ranking | 8 of the wider cell’s 10 successes were on functions the pass never fired on; conditioned, the two land at 1.4% and 0.6%, and Fisher’s minimum is p = 0.22 across the fired-only cells (0.50 across the full-164 ones) | report the rate over the functions the transform actually reached, or report no ordering | | the aggregate fell, so the transform works | the aggregate’s second term is the clean rate of functions the transform never fired on | publish the firing rate per cell, or the drop is unattributable | | near-total deterrence in the o-mvll cell | 81.7% of that cell’s functions exceeded the context window and were scored as failures unseen; 1 of the 30 seen whole came back | print the (window, disassembler, tokenizer) triple and the truncation rate on every cell, recomputed per arm | | a no-op will obviously fail the metric | outcomes identical to clean on all 164 functions, aggregate unchanged from that arm’s 9.15%, and the metric passed it | require a metric a no-op fails, or it certifies nothing | | our negative control settles the point | all six of its firings landed on functions that were already unrecoverable when clean | a null needs power; state when yours has almost none |

Thirty-five papers reported a drop and did not say what it was a drop over. Our own grid reported a ten-to-one gap between two transforms and did not say that four fifths of the wider cell’s successes came from functions the pass had declined to touch. Same defect, wildly different consequence, and both of them invisible to any reader who only has the number.

The next article changes the scope from the binary to the environment it was built in, two arms, the same functions, zero obfuscation in either one, and a gap nobody put there on purpose.

The census is one subfield; the rule it leaves behind is not. Any eval that folds inputs a transform never touched into the same average, without reporting how often it fired, inherits this exact bug.

You build the corpus twice. Once clean, once through the obfuscator. Then you report how far the AI decompiler’s success rate fell.

But the interesting obfuscators are compiler passes, so building that second corpus changes the compiler too. On the corpus below, the compiler switch alone is worth 12.1 points of measured resistance — with no obfuscation anywhere in either arm.

The quantity that moves is re-executable recovery. A neural decompiler reads stripped assembly — machine instructions with the function and variable names thrown away — and writes back C. The metric counts the fraction of functions whose emitted C actually compiles and reproduces the original’s behaviour.

It is the strictest metric in this field, and its drop between clean and obfuscated binaries is what gets published as obfuscation strength. Here it moved 12.1 points on a change nobody had proposed as a treatment.

A controlled comparison isolates the variable you named only if that variable was the only one that moved, and in this field switching obfuscators almost always switches compilers too.

The comparison that looks airtight¶

The field’s standard experiment is a good experiment. Compile a corpus clean. Compile it again through an obfuscator. Run the same decompiler on both. Report the drop. One knob moved, one outcome moved, attribute the delta to the knob. That is what a controlled comparison means, and the instinct behind it is sound, it is how you would measure a drug, a cache policy, or a compiler flag.

It is also, overwhelmingly, what gets published. In the census the first piece in this series walks through, 35 of the 37 papers that report an obfuscation-strength number at all report exactly one bare aggregate scalar for it (37 is the obfuscation-arm subset of 91 full-text-verified papers; those denominators are unpacked in that piece).

The steelman is stronger than it looks, because at the level of author intent it is simply true: the only thing the author touched was the obfuscator. Nobody edited the training set. Nobody swapped the model.

But those obfuscators live inside the compiler, at the intermediate-representation level, which means you obfuscate by building your whole corpus with the toolchain the obfuscator plugs into. That toolchain is usually clang/LLVM. And the model in every cell below, LLM4Decompile v1.5, is trained on GCC output, which is a documented property of its released training corpus rather than an inference from my result. The mechanism has independent attestation too: Decaf, an unrelated system, reports that recompiling its inputs with clang instead of GCC degrades its own GCC-trained reranker, and attributes that to assembly patterns the model was never exposed to. So the clean arm and the obfuscated arm differ by two things, and only one of them is in the paper’s table.

One thing changed, and it was the toolchain¶

So we built the corpus twice, once with GCC, once with clang, both x86 at O2 (the optimization setting most shipped software is built with), neither arm touched by any obfuscation pass, ran the stock published decompiler (LLM4Decompile v1.5) over both with the same prompt, the same 4096-token native context window, the same seed and the same gcc-lenient re-executability scorer, and compared the two arms function by function, pair by pair, an arm built to read as a null.

GCC: 27.5% (41/149). Clang: 15.4% (23/149). A difference of +12.1 points in favour of the toolchain the model was trained on, on binaries with zero obfuscation in them. Both arms here are stock v1.5. The attacker piece in this series reports a clean rate of 18.8% on these same 149 functions from a different checkpoint, the fine-tuned clang-matched control; a clean rate is a property of the attacker, so those two belong to different experiments and cannot be read as a range.

The pairing is the point, so the test respects it. Paired McNemar, a test that ignores every function the two arms agree on and looks only at the ones where they disagree, gives p = 0.0029. The Newcombe paired 95% confidence interval for the difference is [+4.6, +19.6] points, and a bootstrap, resampling the data 20,000 times and reading the spread, which assumes almost nothing about its shape, lands on the same interval to under 0.1 point. Three routes to one answer, and the assumption-light one agrees with the other two.

The model had not been beaten by obfuscation. It had been shown a dialect it never trained on.

Everything about the semantics of those 149 functions is identical across the arms, same source, same behaviour, same optimization level. What differs is register allocation, instruction selection, idiom choice, prologue shape: the accent of the codegen, and that accent is worth twelve points of measured “resistance” that no obfuscator produced and no paper would have attributed to anything else.

The law underneath¶

The general statement is the one that carries outside this field:

A model evaluated on data generated by a different toolchain than its training distribution is measuring distribution shift, not a security boundary.

The word “toolchain” is doing the general work there. It means whatever pipeline turned the world into your model’s input tokens. If the treatment arm’s data came out of a different pipeline than the training data did, the degradation you measure contains a dialect term, and the dialect term is invisible in the table.

The shape is easy to find elsewhere. An OCR system evaluated on documents from a scanner nobody trained on. A speech model whose “noisy” split was re-encoded through a different codec than the clean one. A model eval whose “hard” subset was regenerated by a newer formatter, so the hard split is also the unfamiliar-formatting split. In each case the measurement is real, reproducible, and mis-labelled.

The test is mechanical. List every pipeline stage that produced your treatment arm; cross out the ones that also produced your training data; whatever is left is inside your number. A treatment arm built by a different tool is the most common way that list ends up non-empty.

Three terms, measured separately¶

The fix follows directly: if the environment carries effects, measure the environment’s effects as their own contrasts. We named three, and measured each one as an isolated paired comparison, each with its own zero base, its own interval, and its own contrast.

Compiler provenance is the one above. The second is disassembler format, the model’s actual input is text that a disassembler printed, and two disassemblers print the same instructions with different syntax and layout conventions. Swapping GNU objdump text for llvm-objdump text is worth +5.6 points at n = 162, p = 0.049. The third is prompt fidelity: whether the prompt tells the model which optimization level it is looking at. That is worth +1.9 points, p = 0.58.

The scoreboard follows, with the two rows that lose kept in.

Holm is the correction you owe a family of tests: run three of them and the bar for each one rises, because the chance that something clears 0.05 rises with the count. The format term at 0.049 clears an uncorrected threshold by a hair and does not clear a family-wise one. One discordant function flipping the other way erases it. The prompt term’s interval spans zero, and the paper demoted it in public to a sign-consistent observation, which is all an interval spanning zero will support.

Two demoted rows out of three named terms is the honest yield of this leg. They are printed at full size because both of them lose.

The terms do not sum¶

Add the two measured effects to the base cell and the fourth corner should read 25.3 + 5.6 + 1.9 = 32.8%. It reads 27.2%. The prediction misses by about 5.6 points, and in the paper’s own words the three terms “are not additive shares, and do not sum.”

Here is the full 2 × 2 that says so, four obfuscation-free cells, all on GCC binaries, gcc-lenient scorer, native 4096-token window, x86 at O2, n = 162, with Wilson 95% intervals in brackets (the interval for a proportion when the count is small enough that the textbook formula misbehaves):

Both opt-aware cells come out at the same count, 44 of 162. The format gain that is worth +5.6 points under opt-unaware prompting is worth nothing at all once the prompt names the optimization level, and that is where the missing 5.6 points went.

The format term is +5.6 points and the additivity residual is ~5.6 points, and those are different quantities that happen to coincide. The first is a measured paired difference between two disassembler formats under opt-unaware prompting. The second is arithmetic: predicted corner minus measured corner. And the coincidence is thinner than it looks, because the two 44s are not the same 44 functions. Checked pairwise against the per-function records, the two opt-aware cells share 37 recovered functions and differ on 7 in each direction, so the shortfall equals the vanished term arithmetically rather than because the same functions recovered.

That locates the interaction. It does not explain it. Why conditioning a prompt on optimization level should absorb a disassembler-syntax effect is unexplained here and unexplained in the paper. The four intervals overlap visibly, so the honest label on the interaction is suggestive and unestablished. This section was first drawn as a waterfall, the naive total on the left, three descending steps, one per term, landing on the corner cell, and a waterfall’s whole gestalt is additivity: the reader’s eye completes the sum before it reaches any caption disclaiming one. The arithmetic above forbids that chart, which is why it is not here. In what replaced it the gap is drawn as empty space, because there is no mechanism to fill it with.

And there is a second, larger hole just outside this frame. Our matched corner is 27.2% (44/162, our harness). The published cell it corresponds to, clean re-executability at O2 from GNU objdump text, in the source model’s own harness, is 36.71%, with no confidence interval published. That is about 9.5 points we cannot decompose. Two harnesses measuring what is nominally the same quantity disagree by more than any term this piece measured, which sets the scale for how far a single number from one harness can ever be carried toward a number from another.

We have three candidate causes and no weights for any of them: their exact re-executability scorer, full-object disassembly against our single-symbol disassembly, and their generation stack against ours. Reported open. Also worth stating plainly, since the whole comparison leans on it: identifying their published table’s checkpoint as the release we ran is an inference from matching numbers and matching release dates; the source paper prints no revision string at all.

What this bounds, and what it does not¶

This is one leg, and the leg has walls. One model family at the 6.7B scale, x86 at O2, one re-executability scorer (gcc-lenient), one 4096-token native window, one seed for the factorial cells, and the denominators in the margin note above. Move any of those and every cell has to be recomputed from scratch, the truncation rates in particular are properties of a (window, disassembler, tokenizer) triple, and instances that overflow the window here are excluded from the denominator entirely, which is why 15 functions are missing from the paired 149.

Every recovery number in this piece is re-executability: emitted C that compiles and behaves like the original. Re-executability has its own axis, recovering the shape of a function and recovering a working function are different measurements with different scales, and a structural or control-flow-similarity score from another paper cannot be read across to any cell here.

And the claim is about the instrument: a published deterrence number carries a term nobody named, which is a statement about how the field measures. The attacker in this leg is stock and obfuscation-naive. These numbers bound the instrument, not any shipped product, and a better-resourced attacker only moves them up.

What a controlled comparison has to show¶

The null arm turned out to be the largest, best-attested effect in this leg. Two arms, the same functions, the same model, nothing done to either one except handing them to a different compiler, and it moved the metric further than the obfuscation-strength cells it was built to audit, 35 of whose 37 ship as one bare scalar. Run first rather than last, it would have put a provenance term on every obfuscation cell downstream from the beginning. The 12.1 points were sitting there the whole time, in every paper that ran clean-versus-obfuscated across a toolchain boundary, wearing the obfuscator’s name.

The first piece conditioned the metric on whether the transform had changed the function at all; this one decomposed the environment. The next piece changes the attacker, and the number that had survived every correction so far does not survive that one.

Coverage and provenance hide in the same place: a term sits inside the average and nobody prints it. Any eval that folds inputs a transform never touched into the same average, without reporting how often it fired, inherits this exact bug.

An obfuscation result reads like a property of the binary. Turn the defence on, watch the AI decompiler’s success rate fall to zero, call the code protected.

Then train that same decompiler on the same transforms. It reads 12 to 16% of the very same bytes straight back.

So the zero measured what the attacker had never seen. The headline number describes the attacker at least as much as the defence.

Both halves were measured here on one corpus. A neural decompiler — a transformer that reads stripped assembly, machine instructions with the function and variable names thrown away, and writes back source code, recovered working C from 18.8% of clean functions, and from 0.0% of the 97 functions a constant-encryption pass actually fired on. The comparison was paired, function by function, with a negative control sitting beside it, and every cell in the piece is scored one way throughout: gcc-lenient, at the model’s native 4096-token window.

Measured that way, obfuscation strength is a joint property of two things: what the transform does to the code, and what the attacker happens not to know. Only one of those was in the table.

The cheap objection, that the attacker was simply too small, has a measurable answer, below. The expensive one is that the attacker was never told what had been done to the code. That is the one that landed, and it cost the word hardening in a results table.

The collapse, steelmanned¶

The experiment is easy to state. Take 164 C functions, compile them for x86 at O2 (the optimization setting most shipped software is built with), disassemble them, and ask the model for source that compiles and reproduces the reference behaviour on its test inputs. That strict metric is re-executable recovery, output that merely looks like the original scores zero. On clean binaries this attacker recovers 18.8% of them, 28 of 149.

Then run each transform and re-measure, counting only the functions where the transform actually changed the object’s bytes and the prompt still fit the window. That conditioning is fired-only, and the first piece in this series spends its whole length arguing for it: a transform that never touched a function has no business collecting that function’s clean recovery rate on its behalf.

Constant encryption: 0.0% (0 of 97). Control-flow flattening: 0.0% (0 of 63). Bogus control flow: 0.9% (1 of 106). Those cells are the original seed, 0xC0FFEE; two further held-out seeds reproduce them, with flattening firing on 64 functions instead of 63 in one of them. The scorer is the more permissive of the two re-executability scorers we ran, and the window is the model’s native 4096 tokens throughout.

If you read the first piece, you will notice flattening and bogus control flow have swapped places since its raw aggregates. That swap is itself a finding: at these counts neither ordering is real, and which one you get depends on the regime you measured in. That piece’s fired-only cells sit in a 0-1.4% band against a 9.15% clean anchor, measured with the strict clang-side scorer at an 8192-token window, a separate measurement on a separate axis from these.

Those three denominators come from one corpus of 164 functions, and they differ because the transforms differ. A pass that rewrites every function fires on more of them than a pass that rewrites string literals; a pass that inflates the instruction count pushes more of the surviving functions past the window, where they are scored as failures for reasons that have nothing to do with security. Each cell therefore ends at its own n, and putting a count from one beside a rate from another is its own defect.

Against an 18.8% clean base, that is a collapse to roughly nothing, and it holds up under the checks that usually break this kind of result. The underlying effect survives a paired McNemar test, the standard test for a before/after flip measured on the same items, at p = 2.4e-4, with a placebo config as its negative control. That placebo is a backend-marker pass that perturbs 6 of the 164 objects and leaves every per-function outcome identical to clean; that outcome identity was established in the first article’s clang-strict arm, and what the placebo does here is show that the pipeline does not manufacture a drop when nothing meaningful was done to the code.

So: a large drop, on matched functions, with a paired test and a control. Stopping there ships a deterrence claim with real statistics behind it.

Maybe the attacker was just small¶

That is the first objection anyone sensible raises, and prose cannot answer it. A 6.7B-parameter decompiler failing on obfuscated input is compatible with two very different worlds: one where the transforms genuinely destroy recoverable structure, and one where the model was never big enough to read through anything harder than an unobfuscated function. Those worlds are distinguishable only by making the attacker bigger and holding everything else still.

So we ran the same six configurations, clean, string encryption, constant encryption, control-flow flattening, bogus control flow, and the heavy stack of all three, against three obfuscation-naive attackers at 6.7B, 9B and 22B parameters, same evaluation, same seed 0xC0FFEE. All three are stock LLM4Decompile-v2 Ghidra-refinement checkpoints: one published family at three sizes, each reading a traditional decompiler’s pseudo-C rather than raw disassembly. Holding the evaluation fixed is the whole design, and it is the part that gets quietly dropped in scaling comparisons: change the model and the prompt and the window and the scorer together, and any difference you find belongs to the bundle rather than the parameter count. Here only the weights move. The cell to watch is bogus control flow, because it is the obfuscated configuration with the largest number of functions that both fired and survived the window, the widest denominator in the set, so the least noisy place to look for a scaling effect.

It reads 18.5% [13.2, 25.5] at 6.7B, 15.9% (24 of 151) [10.9, 22.6] at 9B, and 19.9% (30 of 151) [14.3, 26.9] at 22B, brackets being Wilson 95% intervals, the interval you get for a proportion when the count is small enough that the textbook formula misbehaves. A 3.3x increase in parameters moves the middle rung down and the top rung barely above the bottom. Every interval overlaps every other interval.

The obvious escape is a ceiling effect, perhaps the bigger models were not actually better at this task at all, so of course obfuscated recovery did not move. The clean bases refuse it: 32.5%, 33.8%, 31.8% across the three scales, already flat before any transform enters the picture. Nothing in this ladder rewards parameters, on clean input or obfuscated.

Two numbers in this piece now sit in obvious tension. Section one reads bogus control flow at 0.9%; this section reads it at 18.5% at the same 6.7B parameter count. Both are correct, and each belongs to its own evaluation family. The ladder’s rungs are stock Ghidra-refinement models reading a traditional decompiler’s pseudo-C; the collapse leg’s attacker is the fine-tuned clang-matched control reading raw disassembly of clang-built objects. Two conditions differ at once, the checkpoint and the input representation, which is enough to establish that the two clean bases do not belong on one axis and not enough to say which of the two moves the number: 32.5% here, about 1.7x the 18.8% there, over 151 functions rather than 149. That 1.7x is left undecomposed. The ladder’s three rungs are comparable to each other and to nothing else in this piece, which is precisely why it can answer the scaling objection and no other question.

The honest caption on that ladder avoids the word “flat.” Three points with intervals that wide are compatible with a rising curve, a falling curve and a level one; a reader whose intuition lands inside them has not been refuted. What the ladder does establish is narrower and sufficient: within this range, on this evaluation, parameter count does not buy read-through.

Scale is not the variable this leg tests.

Tell the attacker what you did¶

Kerckhoffs published the principle in 1883 for ciphers, and it transfers without modification: a system has to stay secure when the adversary knows exactly how it works. Every number in section one was measured against an adversary who did not. The naive attacker had seen ordinary compiled code and had never seen a flattened control-flow graph or an encrypted constant pool in training. Its 0.0% is a measurement of that gap as much as of the transform.

There is a mechanical reason to expect this to matter, and it is the least surprising thing in the article. These transforms are compiler passes. A pass emits the same idioms every time it runs, the same dispatcher shape for a flattened graph, the same decrypt-then-use sequence around a hidden constant, because that is what a pass is. Those idioms are precisely the kind of regularity a sequence model absorbs from examples. Withholding them from the attacker’s training set is a choice about the experiment, and the experiment then reports the consequences of that choice as a property of the binary.

So we removed the gap. Fine-tune the attacker on the same transform family, then re-run the identical evaluation, same binaries, same functions, same window, same scorer, same 18.8% clean baseline. Only the attacker’s training distribution changed.

Constant encryption: 14.4% (14 of 97) [8.8, 22.8]. Control-flow flattening: 14.3-15.6%. Bogus control flow: 12.3% (13 of 106) [7.3, 19.9]. The heavy composite, all transforms stacked: 16.0%.

Read the constant-encryption cell twice. It is 0 of 97 for the naive attacker and 14 of 97 for the obfuscation-aware one. Same denominator, same functions, same bytes on disk. Nothing about the binary changed between those two numbers.