thepragmaticquant — fired-not-fed

Nobody published the denominator

Thu, 30 Jul 2026 00:00:00 GMT

TL;DR — The models here read compiled code and write source back, and the number that scores them cannot be recomputed by anyone who was not in the room. When a paper claims an obfuscator defeats an AI code-reverser, it publishes one number: how far the reverser’s success rate fell. Nobody publishes what that rate was computed over. Across 37 papers reporting such a number, 1 conditions it on whether the transform actually fired on the function being scored — and that one scores disassembly rather than re-executability, which leaves nothing on the axis these numbers live on — and 0 stratify it by context-window truncation. Condition our own grid on firing and a two-transform ranking we had already written down collapses to noise: on the strict clang-side scorer at an 8192-token window, Fisher’s exact test bottoms out at p = 0.22 across the fired-only cells, nowhere near separating, and no ordering survives in either direction. Then a control configuration whose per-function outcomes are identical to the clean build passes the field’s standard metric as a working obfuscator. Preprint: doi:10.5281/zenodo.21682266.

You sell a compiled program. The buyer owns the machine it runs on, so they can take it apart at leisure.

Obfuscation is the defence: a compiler pass that leaves the program’s behaviour alone while making its compiled form harder to read back. Every paper measuring how well it holds against AI code-reversers reports one number — how far the reverser’s success rate fell.

That number averages two populations that behave nothing alike. There are the functions the transform changed. And there are the functions it never touched, which recover exactly as well as they did before anyone tried to protect them, and get counted into the obfuscated result anyway.

So the headline can move because the defence worked, or because the transform covered less ground. Buyers choose on those published numbers, because those numbers are the only evidence anyone has.

Nothing that gets published lets a reader tell the two cases apart.

The defect is arithmetic before it is anything about binaries. This series measures it in neural decompilation because that is a setting where each missing term can still be put back and counted: whether the transform fired, whether the model read the function whole, which toolchain and which disassembler rendered the text, and what the attacker had already seen.

The one number everybody reports¶

The metric deserves a steelman, because the arithmetic behind it is sound.

A neural decompiler is a transformer that reads stripped assembly — machine instructions with the function and variable names thrown away — and emits compilable C. The obfuscation passes it is tested against are the ones above: flattening control flow into a dispatch loop, injecting branches that never execute, encrypting strings and constants so they materialize only at runtime. The attacker downloads the shipped binary and runs a decompiler over it, so the field’s measurement is the natural one: decompile clean, decompile obfuscated, report how far recovery fell.

Recovery is defined strictly. A function counts as recovered when the emitted C compiles and reproduces the reference behaviour on its test inputs — output that merely looks like the original scores zero. A second family of metrics scores the shape of what came back, usually the recovered control-flow graph’s similarity to the original. Different question, different axis; every number here stays on the re-executability axis, and any pair that needs a bridge between the two gets one in the text.

So: one scalar, strictly defined, cheap, comparable across papers, shaped exactly like the accuracy-drop-under-attack number adversarial machine learning has reported for a decade. It is a reasonable thing to publish. What goes unpublished is the population it was computed over.

What thirty-seven papers publish¶

We built a census to find out whether that omission was ours or the field’s. The corpus is 96 papers after deduplication, 96 papers, unrelated to the 96 configurations of our own grid further down. Of those, 91 were verified against their full text, and 37 report an obfuscation-strength number of the kind above.

We then scored each of those 37 for the four things a reader needs to recompute the number. Conditioning the rate on whether the transform actually fired on the function being scored, fired meaning the compiled object’s bytes for that function change relative to the clean build: 1 of 37. Stratifying it by truncation, these models read a fixed number of tokens, and anything longer is cut off and scored as a failure, which the fifth section takes apart, so a reader can tell how many instances the model never saw whole: 0 of 37. Decomposing the drop by toolchain provenance, meaning which compiler built the binaries, since two compilers turn identical source into visibly different assembly: 0 of 37. Reporting a bare aggregate with none of the above: 35 of 37.

Every published paper as a row, every plank a reader needs as a column. The four columns that would make the number recomputable are empty; the one that makes it un-auditable is full. Scored single-pass over 91 full-text papers, adversarially re-verified on 70 with 233 of 235 cells upheld; recomputable with the deposit's prevalence_tally.py.

data table

plank	what it requires	k/n	rate	Wilson 95%
A	fired-only / coverage-conditioned	1/91	1.1%	[0.2, 6.0]
B	truncation-stratified	3/91	3.3%	[1.1, 9.2]
C	confidence intervals (6 firmly + 6 weakly)	12/91	13.2%	[7.7, 21.6]
D	provenance decomposition	2/91	2.2%	[0.6, 7.7]
BS	bare aggregate scalar (the absence of a plank)	83/91	91.2%	[83.6, 95.5]
18 of 364 plank cells met (A+B+C+D over 91 papers x 4 planks). The lone A mark is DisasLLM, whose firing-conditioned cell scores disassembly precision/recall/F1, not re-executability recovery, so it scores nothing on the axis that matters here.

The single positive in the firing-conditioned column carries a disqualifier, and the disqualifier travels with the count everywhere it is quoted. That paper, DisasLLM, named in its row of the scored table the deposit ships, does condition on firing, but its firing-conditioned cell is a disassembly precision/recall/F1 row rather than a re-executability recovery cell. On the axis this piece is about, the count is zero.

The corpus was scored single-pass, then adversarially re-verified on 70 of the 96 papers, with 233 of 235 re-examined cells upheld. Only two papers were double-scored and the category distribution is degenerate, so no inter-annotator statistic is computable.

The finding is deliberately unexciting: nobody is cheating. Every one of those 35 numbers appears to be correctly computed over whatever population its authors used. There is simply no way for a reader to learn what that population was, no way to compare two papers’ numbers, no way to tell a strong transform from a lucky denominator. The deposit ships the scored table and the tally script, so the counts can be recomputed rather than trusted.

The functions the transform skipped¶

Here is the mechanism that makes the scalar un-auditable.

Firing is a per-function property. A transform that flattens control flow needs control flow to flatten; a pass that encrypts string literals needs string literals; a pass gated on a size heuristic skips whatever falls below the threshold. So on any real corpus some functions come out changed and some come out byte-identical to the clean build. Both are handed to the decompiler, and both are counted in one average:

text

aggregate recovery
  =  share of functions the transform fired on
       x  recovery among those functions
  +  share it never fired on
       x  recovery among those        <-- which is just the clean rate

The first term is the only one that answers the question anyone asked. The second is clean-rate contamination: functions the transform never modified, still recovering at their pre-obfuscation rate, arriving in the obfuscated column as though the transform had been tested on them. Push the firing rate down and the aggregate drifts toward the clean rate no matter how effective the transform is on the functions it does reach.

Put round numbers on it. A transform fires on 20 of 100 functions and recovers nothing at all on those 20, total defeat on everything it touched. The other 80 were never modified, so they recover at the clean rate of 10%, which is 8 functions. The published aggregate is 8%, down from 10%, and it reads as a transform that barely helped. The 20 it actually reached were annihilated.

Each configuration's 164 functions, marked by what happened. The pale never-fired cells are the ghosts: counted at the clean rate, never touched by the transform. Firing ranges from 6 of 164 to all 164 across configurations, so the dilution is per-cell and no after-the-fact correction recovers it. Stock v1.5, clang-strict scorer, 8192-token window, x86/O2, seed 0xC0FFEE.

data table

config	transform	fired (byte-diff) /164	ghosts	analysed n	fired-only k/n	raw aggregate	recoveries that are ghosts
cfg 16	backend-marker (placebo)	6/164	158	6	0/6 = 0.0%	12/164 = 7.3%	12 of 12
cfg 1	string_encryption	29/164	135	25	0/29 = 0.0%	12/164 = 7.3%	12 of 12
cfg 4	flatten_cfg	139/164	25	130	2/139 = 1.4%	10/164 = 6.1%	8 of 10
cfg 2	constant_encryption	157/164	7	151	0/157 = 0.0%	4/164 = 2.4%	4 of 4
cfg 8	bogus_control_flow	164/164	0	158	1/164 = 0.6%	1/164 = 0.6%	0 of 1
cfg 14	heavy composite	164/164	0	122	0/164 = 0.0%	0/164 = 0.0%	0 of 0
Clean baseline, same conditions: 12/164 = 7.3% raw, 12/152 = 7.9% truncation-excluded. Interval-width pair: 0/29 [0.0, 11.7] vs 0/157 [0.0, 2.4], the same 0.0% carrying very different amounts of information.

The never-fired population is wildly heterogeneous. Some configurations in our grid fire on every function and have no ghosts at all; sparse, heuristic-gated ones fire on a small minority and are almost entirely ghosts. The distortion therefore varies per cell, no rule of thumb corrects for it afterwards, and only the person who ran the transform can publish it.

Conditioning on firing has a cost, and the cost is honest: the denominator shrinks and the confidence interval around the rate, the range the true value plausibly occupies, given how few functions were counted, widens with it. Zero successes out of two dozen functions and zero out of a hundred and fifty carry the same point estimate and very different amounts of information. The wide interval tells you the experiment was small, which the aggregate hid by padding the denominator.

The statistical idea is old, and it came from mutation testing, which solved the same problem decades ago with the Covered Mutation Score: score a mutant only over the tests that actually reach it, because a mutant no test covers tells you nothing about the suite. What is new here is the application, nobody had carried the covered-only discipline across into obfuscation-strength claims, where the uncovered population is a majority in some configurations and empty in others. Adversarial machine learning, the field whose accuracy-drop-under-attack number this one is shaped like, reached the same discipline from the other side: its evaluation-hygiene canon conditions attack success on the perturbations that were successfully generated.

The same fix, pointed at our own grid, cost us a result we had already written down.

A word on structure, because the rest leans on it: the grid is a 96-configuration factorial of our own obfuscation toggles, each run over the same 164 functions, 95 obfuscated configurations plus the clean baseline, which is itself a point inside the grid, so every grid-wide statistic below is computed over 95. A cell is one configuration’s number together with the population it was computed over; an arm or leg is a whole sub-experiment with its own model, scorer and window.

Every cell in this section comes from one arm: the stock released LLM4Decompile v1.5 over 164 functions per configuration, scored with the strict clang-side re-executability scorer at an 8192-token context window, the amount of disassembly the model can read at once, on x86 binaries built at O2, the setting most shipped software uses, seed 0xC0FFEE. Read the raw aggregates and a clean story is sitting right there: control-flow flattening leaves recovery at 6.1% while bogus control flow crushes it to 0.6%. Ten to one. Injecting dead branches looks dramatically stronger than restructuring control flow, which is counterintuitive, publishable, and exactly the kind of result that makes a grid worth running. We wrote it into the table as a ranking.

Conditioning both cells on firing collapses them into each other. Both land in the 0–1.4% band, and Fisher’s exact test, the right test when counts are small enough that a chi-squared approximation misbehaves, separates nothing.

Which number you quote depends on the denominator you tested, so both belong in print. Across the full-164 cells, each transform’s fired-only successes carried over the whole corpus, every pairwise contrast returns p > 0.3, with a minimum of 0.50. Across the fired-only cells, where each transform carries its own denominator, the minimum falls to p = 0.22: control-flow flattening’s 2 of 139 against constant encryption’s 0 of 157, the third transform in the same collapsed band and the pair that comes nearest to separating anything. Attaching the roomier full-N figure to a fired-only cell is precisely the splice this piece exists to name, so it does not happen here. Neither figure buys an ordering. At two successes between the two cells, 0.22 is what the arithmetic returns when there is nothing there to find.

The ranking on the left is the raw aggregate; the floor on the right is the same transforms scored only over the functions they fired on. Conditioning collapses the ten-to-one gap into overlapping noise. Stock v1.5, clang-strict scorer, 8192-token window, x86/O2, seed 0xC0FFEE; Wilson 95% throughout.

data table

transform	raw k/n	raw %	fired-only k/n	fired-only %	full-N k/n
string_encryption	12/164	7.3%	0/29	0.0% [0.0, 11.7]	not published
flatten_cfg (CFF)	10/164	6.1%	2/139	1.4% [0.4, 5.1]	2/164
constant_encryption	4/164	2.4%	0/157	0.0% [0.0, 2.4]	0/164
bogus_control_flow (BCF)	1/164	0.6%	1/164	0.6% [0.1, 3.4]	1/164
clean (comparator)	12/164	7.3%	12/152	7.9% [4.6, 13.3]	,
Pairwise Fisher exact, two-sided: minimum p = 0.22 on the fired-only denominators (CFF 2/139 vs const 0/157), 0.50 on full-N. No pairwise contrast separates. “All p > 0.3” is a full-N statement and is not printed beside fired-only cells.

The dissolution runs in the direction that costs the most, and the arithmetic is small enough to check by hand. Bogus control flow fires on all 164 functions, so its 0.6% was already a fired-only number and does not move. Control-flow flattening fires on 139 and skips 25, and 8 of its 10 apparent successes sit in those 25, functions the pass never touched, recovering at 32% against this arm’s 7.3% clean rate, because they were the easy ones it declined. Scored over the 139 it reached, flattening reads 1.4% (2/139) against bogus control flow’s 0.6% (1/164).

The collapse licenses one sentence and no more. It does not establish that the two transforms are equivalent: at the success counts these cells contain, the data pays for exactly no ordering established at this n, and an equivalence claim would need a powered equivalence test this cell cannot support.

We killed the ranking. A protocol that only ever indicts other people’s numbers is a marketing document.

A no-op that passes¶

The ranking dissolution is a corollary. This is the wall.

Somewhere in the grid sits a backend-marker pass: it writes a short tag into the object file recording which code generator produced it, a build stamp, and changes nothing about the program’s structure, its control flow, or its data. Its per-function results are worth stating exactly, because the exactness is the entire argument.

They come from a different checkpoint than the ranking above, the fine-tuned clang-native 6.7B control attacker rather than the stock release, on the same strict clang-side scorer, the same 8192-token window, the same 164 functions at x86/O2, the same seed. Two checkpoints, so the two sections’ rates cannot be compared with each other.

It fires on 6 of the 164 functions: six compiled objects whose bytes differ from the clean build. The other 158 are byte-identical, every byte, and you can put the two hex dumps side by side and look. Its fired-only recovery is 0 of 6. And its per-function outcome triples, recovered, compiled, truncated, are identical to the clean build on all 164 functions, with 0 differences and the same 15 recovered functions. Its aggregate recovery is therefore unchanged from that arm’s clean rate of 9.15%.

The placebo config's object next to the clean object for one function: byte-for-byte identical, same hash. Across all 164 functions the outcome differs in zero cases, yet the config's aggregate is the clean rate of 9.15% and the field's metric records it as a working obfuscator. It is outcome-inert, not byte-inert, firing is a byte diff, and it does perturb 6 of 164 objects; what is identical is the outcome. D1C control attacker, clang-strict scorer, 8192-token window, x86/O2.

data table

claim	as audited
the config hardens the binary	byte diff on 6 of 164 objects (func ids 5, 26, 87, 147, 151, 158); the other 158 are byte-identical to clean
recovery under the config is suppressed	per-function (reexec, compiled, truncated) identical to clean on all 164, 0 differences, same 15 recovered func ids, the number is the clean cell, 9.15% (15/164)
it is a negative control for the metric	fired-only 0/6 [0.0, 39.0]; clean on those same 6 is also 0/6, all firings land on clean-non-recoverable functions, so it shows the metric’s blindness, not a powered null
displayed object (func41)	1,016 bytes; sha256 clean == placebo == 41bacddf…0df647; equal on all 64 hex rows; cmp of all 164 objects gives diffset 158

Two things about that 9.15% before anyone quotes it. It is low because of the scorer: score the same checkpoint on the same 164 clean functions with the permissive gcc-side scorer and the clean rate reads 15.9% (26/164), same model, same window, same functions, one scoring rule apart, which is why no clean rate in this series travels without the scorer that produced it. And the pass is aggregate-identical under the stock checkpoint too, the same 12 functions recovering with and without it, so the blindness below survives a change of attacker.

The field’s standard metric reports no meaningful drop here, which is correct, and which is the problem: it had no way to distinguish “this transform does nothing” from “this transform does nothing detectable by this attacker on this corpus” from “this transform was never applied to most of the corpus.” All three read as the same scalar. A metric that cannot fail a no-op cannot certify a transform.

What the configuration demonstrates is that blindness, and only that. All six functions it perturbs were already unrecoverable in the clean build, so the cell has close to no statistical power to detect an effect if one existed. It cannot serve as the protocol’s load-bearing negative control; sold as a powered null it would invert under the first competent reviewer.

The wall the fix does not reach¶

Conditioning on firing fixes coverage. It does nothing whatsoever about the second confound, and the second confound is arguably worse because it is invisible from inside the metric.

Neural decompilers have a context window. Feed one a disassembly listing longer than that window and the input is cut off; the model answers on a fragment, fails to produce compilable output, and the harness scores the instance as a failure. Which it is, a failure of the measurement apparatus, telling you only that the listing was long. Obfuscation passes that duplicate blocks or inject dispatch loops make listings much longer, so they push instances over the window and harvest failures the model never had a chance at. What moved was the truncation rate.

The scale is larger than it sounds. In the o-mvll arm of our own grid, the same fine-tuned control attacker, the same strict clang-side scorer and the same 8192-token window as above, reading llvm-objdump’s rendering of the disassembly, exactly 1 of 164 functions came back working. As an aggregate that is roughly a 99% failure rate, and the field’s vocabulary would call it near-total deterrence. Stratified, it is 81.7% truncation (134 of 164) plus a fired-only recovery of 3.3% (1 of 30) on the functions the model saw whole, whose 95% interval of [0.6, 16.7] comfortably contains that arm’s own 9.15% clean rate.

Prompt length against the context window. Functions past the rule are cut off and scored as recovery failures, so a truncation rate reads as deterrence. Under a faithful OLLVM-family compile, 84.8% of bogus-control-flow functions at O0 overflow the window, which is the regime a published 9.76% recovery cell was measured in, though that rate itself was never reported. Truncation is a property of the (window, disassembler, tokenizer) triple, never of the transform.

data table

arm	n	truncated (over window)	non-truncated	recovered	rate	window / disassembler
Pluto/OLLVM, BCF at O0	164	139 (84.8%)	25	,	,	4,096 native / GNU objdump
o-mvll (D1C control)	164	134 (81.7%)	30	1	3.3% [0.6, 16.7]	8,192 / llvm-objdump
o-mvll clean comparator	164	0	164	15	9.15% [5.6, 14.5]	8,192 / llvm-objdump
Per-cell truncation, Pluto family: bogus control flow overflows 139/164 (84.8%) at O0, longest prompt 32,450 tokens; control-flow flattening 22/164 (13.4%); clean 1/164. The 134 truncated o-mvll items are censored at the 6,135-token budget, their true lengths are unmeasured, so drawing them as a spread would fabricate a distribution.

And a truncation rate is not a property of the transform. It is a property of the (window, disassembler, tokenizer) triple: the same configuration, disassembled by a different tool or tokenized by a different model, truncates at a different rate, and at a smaller window it truncates more. So a truncation rate cannot be carried across arms or rescaled between windows, it has to be recomputed inside each arm and printed on each cell, in the same breath as the recovery number it explains. Zero of the 37 papers do this. Neither did we, in the first cut of our own grid.

The two confounds compound in a way worth naming. Coverage dilution pulls the aggregate up toward the clean rate by padding it with untested functions; truncation pushes it down by scoring unseen functions as failures. Both are population effects, both are invisible in the scalar, and they do not cancel in any predictable ratio.

We can put a ratio on the dilution, what fraction of a configuration’s aggregate recovery comes from functions the transform never fired on, and across the 95 obfuscated configurations it is bimodal, two piles at the ends of the axis. 48 score zero: they fire on all 164 functions, so there is no ghost population to dilute anything, and that bucket absorbs the configurations where nothing recovered at all and the ratio is a 0/0 the code prints as zero. 43 score close to 1.0, meaning almost the entire aggregate is clean-rate contamination. Four sit mid-band near 0.47, and they are the cells whose only transform rewrites arithmetic into mixed boolean-arithmetic expressions, the one corner of the grid where the never-fired population is neither absent nor overwhelming. So the scalar diagnoses whether never-fired functions exist, and against an attacker crushed to near zero on the functions it does reach, that is the whole of its range.

How much of each configuration's apparent effect is coverage dilution rather than hardening, over the 95 obfuscated configs. The scalar is bimodal: near 1.0 (almost all apparent effect is ghosts) or near 0 (the transform fired everywhere). But 44 of the 48 low values are undefined, zero recovery over a zero denominator, a guard value, not a measurement, drawn grey so they do not read as 'no dilution'. Three configs sit at exactly the 9.15% clean rate while recovering nothing on every function they fired on: the mirage made numeric.

data table

state	count (of 95)	what it is
~1.00	43	fired-only ~0% while never-fired functions recover at the clean rate, nearly all apparent recovery is a coverage artifact
~0.47	4	mid-band; the near-MBA-only configs, named individually in the figure
measured 0.0	4	true zeros: the transform fired on all 164, so there is nothing to dilute
undefined	44	aggregate recovery is 0.0%, so the ratio is 0/0, a guard value, drawn grey; the scalar does not exist for these
Among the 51 configs where the scalar is defined it is positive iff at least four functions never fired (n_fired ≤ 160), it reports whether ghosts exist, not transform strength. Scalar = (aggregate − fired-only) / aggregate. D1C control, clang-strict, 8192-token window, x86/O2.

Which leaves the denominators themselves, the mundane failure mode that eats the most numbers. Four of them run through this series:

164, the grid corpus: every function, in every configuration.
149, the matched non-truncated subset: functions whose disassembly fits a 4096-token window on both sides of the next article’s compiler comparison. Fifteen truncate and are dropped from both arms, a different fifteen from the fifteen the placebo section recovers, sharing exactly one member. There is one 149, not two: the two legs that use it were checked function by function and carry the same set.
151, the fired, non-truncated population of one obfuscated cell in the third article’s scaling ladder, at that arm’s own window and prompt. It sits inside 164 and inside 162, but 7 of its functions fall outside the 149: the two overlap without either containing the other.
162, the grid minus the two functions that truncate in the disassembler-format comparison.

None is a superset the obfuscated cells are drawn from; each obfuscated cell has its own fired, non-truncated population. Put flattening’s 2/139 beside bogus control flow’s 1/164 and both rates are honest; put either beside the 10/164 aggregate that produced the ranking and you have drawn a trend across three different populations.

The four denominators this piece and its siblings use. Recovery was never computed over the same population twice, and the population was never printed; this is the reference every rate resolves against.
n	what it counts	arm / scorer / window	drops, and why	what it is NOT nested with
164	the grid corpus, every function in every configuration	all grid legs; scorer and window vary by leg	nothing; it is the base population	the census (96/91/37) and ExeBench (489/473), which are different populations entirely
149	the shared matched non-truncated subset, one construction, reached by two legs that carry the identical func ids	Article 2’s stock-v1.5 provenance leg and Article 3’s naive/Kerckhoffs legs; gcc-lenient; native 4096	the 15 that truncate under clang at 4096; {147, 160} truncate under gcc too and are a subset of those 15	151, they only overlap: 7 of the 151 fall outside the 149 and 5 of the 149 fall outside the 151, so neither contains the other. There is no second 149.
151	the fired, non-truncated population of one obfuscated cell in Article 3’s scaling ladder	Ghidra-Ref v2 (6.7B/9B/22B), obfuscation-naive; gcc-lenient; native 4096	13 functions truncated at that arm’s window	the 149 (overlap only, above); its rates belong to a third denominator family (151/132/98) and must not mix with the 149 or 164 legs
162	the grid minus the two functions that truncate in every disassembler-format arm	Article 2 factorial; gcc-lenient; native 4096	func_id 147 and 160 only	nothing on sets (149 is nested inside 162, 162 inside 164), but its RATE, 44/162 = 27.2%, is the matched corner, not to be printed as disagreeing with 41/149 = 27.5%, the same arm on the paired subset

What this does not show¶

The limits are specific, and three of them are load-bearing.

No published number here is disputed. The census makes no claim that any of the 35 bare aggregates is arithmetically wrong. What it claims is that the presentation is un-auditable: the population, the firing rate and the truncation rate are missing, so a reader cannot recompute the cell or compare it with another paper’s. That is a reporting finding, not an allegation of arithmetic error.

The census’s own reliability is limited. Single-pass scoring, adversarially re-verified on 70 of 96 papers, 233 of 235 re-examined cells upheld, and no computable inter-annotator statistic. That is stronger than one pass alone and weaker than validated dual annotation, and the paper says so in those words.

Every attacker result above is scoped, and the scope is narrow. These are static, client-side measurements against two 6.7B decompiler checkpoints on 164 functions, one scorer and one window at a time, single-shot: no tool use, no iteration, no human in the loop. Every number here is a floor an attacker starts from. A better-resourced attacker is the subject of the third piece in this series, and it moves them substantially. Nothing here says any shipping product is broken. Client-side hardening is deterrence, never a guarantee.

Two smaller limits. The backend-marker configuration’s null is unpowered, all six of its firings landed on functions that were already unrecoverable when clean, so it evidences the metric’s blindness and nothing about the transform. And the first minted version of the deposit calls that configuration “byte-inert,” one page from the definition of firing as a byte difference; the pass changes the bytes of six functions and leaves the outcomes alone, so the correct word is outcome-inert, and the correction goes into the next version.

One bridge, rather than a limit. Both arms here run the strict clang-side scorer at an 8192-token window; the legs in the next two articles run a more permissive gcc-side scorer at the model’s native 4096-token window, which is a different measurement and cannot go on the same axis without saying so out loud. It is also why the two transforms above trade places between articles: at these counts the ordering is a property of the measurement regime.

What an obfuscation number has to carry¶

| what read fine | what was actually true | the rule it became | |, |, |, | | a tenfold gap between two transforms is a ranking | 8 of the wider cell’s 10 successes were on functions the pass never fired on; conditioned, the two land at 1.4% and 0.6%, and Fisher’s minimum is p = 0.22 across the fired-only cells (0.50 across the full-164 ones) | report the rate over the functions the transform actually reached, or report no ordering | | the aggregate fell, so the transform works | the aggregate’s second term is the clean rate of functions the transform never fired on | publish the firing rate per cell, or the drop is unattributable | | near-total deterrence in the o-mvll cell | 81.7% of that cell’s functions exceeded the context window and were scored as failures unseen; 1 of the 30 seen whole came back | print the (window, disassembler, tokenizer) triple and the truncation rate on every cell, recomputed per arm | | a no-op will obviously fail the metric | outcomes identical to clean on all 164 functions, aggregate unchanged from that arm’s 9.15%, and the metric passed it | require a metric a no-op fails, or it certifies nothing | | our negative control settles the point | all six of its firings landed on functions that were already unrecoverable when clean | a null needs power; state when yours has almost none |

Thirty-five papers reported a drop and did not say what it was a drop over. Our own grid reported a ten-to-one gap between two transforms and did not say that four fifths of the wider cell’s successes came from functions the pass had declined to touch. Same defect, wildly different consequence, and both of them invisible to any reader who only has the number.

The next article changes the scope from the binary to the environment it was built in, two arms, the same functions, zero obfuscation in either one, and a gap nobody put there on purpose.

The census is one subfield; the rule it leaves behind is not. Any eval that folds inputs a transform never touched into the same average, without reporting how often it fired, inherits this exact bug.

The compiler did it, and my obfuscation took the credit

Thu, 30 Jul 2026 00:00:00 GMT

TL;DR — Same 149 matched functions, the stock released LLM4Decompile v1.5, one scorer for both, no obfuscation in either arm: GCC-built binaries recover at 27.5% and clang-built binaries at 15.4% (paired McNemar p = 0.0029, gcc-lenient scorer, native 4096-token window). A published obfuscation-strength number can carry a double-digit toolchain term it never names. The three-term decomposition built to separate those effects then misses its own predicted corner by ~5.6 points, and the residual is printed open and unexplained. Preprint: doi:10.5281/zenodo.21682266.

You build the corpus twice. Once clean, once through the obfuscator. Then you report how far the AI decompiler’s success rate fell.

But the interesting obfuscators are compiler passes, so building that second corpus changes the compiler too. On the corpus below, the compiler switch alone is worth 12.1 points of measured resistance — with no obfuscation anywhere in either arm.

The quantity that moves is re-executable recovery. A neural decompiler reads stripped assembly — machine instructions with the function and variable names thrown away — and writes back C. The metric counts the fraction of functions whose emitted C actually compiles and reproduces the original’s behaviour.

It is the strictest metric in this field, and its drop between clean and obfuscated binaries is what gets published as obfuscation strength. Here it moved 12.1 points on a change nobody had proposed as a treatment.

A controlled comparison isolates the variable you named only if that variable was the only one that moved, and in this field switching obfuscators almost always switches compilers too.

The comparison that looks airtight¶

The field’s standard experiment is a good experiment. Compile a corpus clean. Compile it again through an obfuscator. Run the same decompiler on both. Report the drop. One knob moved, one outcome moved, attribute the delta to the knob. That is what a controlled comparison means, and the instinct behind it is sound, it is how you would measure a drug, a cache policy, or a compiler flag.

It is also, overwhelmingly, what gets published. In the census the first piece in this series walks through, 35 of the 37 papers that report an obfuscation-strength number at all report exactly one bare aggregate scalar for it (37 is the obfuscation-arm subset of 91 full-text-verified papers; those denominators are unpacked in that piece).

The steelman is stronger than it looks, because at the level of author intent it is simply true: the only thing the author touched was the obfuscator. Nobody edited the training set. Nobody swapped the model.

But those obfuscators live inside the compiler, at the intermediate-representation level, which means you obfuscate by building your whole corpus with the toolchain the obfuscator plugs into. That toolchain is usually clang/LLVM. And the model in every cell below, LLM4Decompile v1.5, is trained on GCC output, which is a documented property of its released training corpus rather than an inference from my result. The mechanism has independent attestation too: Decaf, an unrelated system, reports that recompiling its inputs with clang instead of GCC degrades its own GCC-trained reranker, and attributes that to assembly patterns the model was never exposed to. So the clean arm and the obfuscated arm differ by two things, and only one of them is in the paper’s table.

One thing changed, and it was the toolchain¶

One function, one source, two compilers. This is func_id 0, one of the 26 discordant rows where the gcc build re-executes and the clang build does not, under stock LLM4Decompile v1.5, the gcc-lenient scorer and the native 4096-token window, x86 at O2. The same 27 gcc instructions become 33 under clang, and a reader cannot tell which listing an obfuscator touched, because neither was touched.

data table

measured on func_id 0	gcc	clang	difference
instructions in `<func0>`	27	33	+6 under clang
.text extent (bytes)	92	116	+24 under clang
object file (bytes)	1512	1352	-160 under clang, sections and padding, not a code-size measure
prompt length (tokens)	388	445	+57; both well under the 4096 window, neither truncated
re-executes (gcc-lenient)	yes	no	the flip this figure exists to show
Excerpt is func0 only, one function’s .text out of the 164-function object, nothing inside it elided; offsets, mnemonics and operands are verbatim from the committed listing. Neither arm is obfuscated.

So we built the corpus twice, once with GCC, once with clang, both x86 at O2 (the optimization setting most shipped software is built with), neither arm touched by any obfuscation pass, ran the stock published decompiler (LLM4Decompile v1.5) over both with the same prompt, the same 4096-token native context window, the same seed and the same gcc-lenient re-executability scorer, and compared the two arms function by function, pair by pair, an arm built to read as a null.

GCC: 27.5% (41/149). Clang: 15.4% (23/149). A difference of +12.1 points in favour of the toolchain the model was trained on, on binaries with zero obfuscation in them. Both arms here are stock v1.5. The attacker piece in this series reports a clean rate of 18.8% on these same 149 functions from a different checkpoint, the fine-tuned clang-matched control; a clean rate is a property of the attacker, so those two belong to different experiments and cannot be read as a range.

Four denominators across this series, and they are not nested the way you would guess. 164 is the grid corpus. 149 is the shared matched non-truncated subset, the functions whose disassembly fit the window in both compiler arms, with 15 dropped for truncating in either one. The attacker piece’s clean base sits on this same 149; the two sets were checked function by function and are identical, so there is one 149 in this series rather than two constructions that happen to collide. 162 is the grid minus 2 truncated instances, and it is the denominator for the factorial cells below. 151 is the scaling ladder’s bogus-control-flow cell in the attacker piece, fired and non-truncated at that arm’s window. It nests inside 164 and inside 162, but 7 of its functions fall outside the 149, so 151 and 149 overlap with neither one containing the other. The 15 dropped here are also a different 15 from the 15 recovered functions in the placebo section of the first piece. Putting a count from one of these on an axis with a rate from another is a defect all by itself.

The pairing is the point, so the test respects it. Paired McNemar, a test that ignores every function the two arms agree on and looks only at the ones where they disagree, gives p = 0.0029. The Newcombe paired 95% confidence interval for the difference is [+4.6, +19.6] points, and a bootstrap, resampling the data 20,000 times and reading the spread, which assumes almost nothing about its shape, lands on the same interval to under 0.1 point. Three routes to one answer, and the assumption-light one agrees with the other two.

The hero contrast, drawn thread by thread. Only the 34 discordant threads, 26 favouring gcc, 8 favouring clang, carry the whole paired result; the 115 concordant threads contribute nothing by design. Under stock LLM4Decompile v1.5, the gcc-lenient scorer and the native 4096-token window, the gcc arm recovers 41/149 and the clang arm 23/149, a paired +12.1 points (McNemar p = 0.0029, Newcombe [4.6, 19.6]) on functions no obfuscator ever touched.

data table

arm	k/n	rate	Wilson 95%
gcc binaries	41/149	27.5%	[21.0, 35.2]
clang binaries	23/149	15.4%	[10.5, 22.1]
paired 2×2 cell	meaning	count
a	both arms re-execute	15
b	gcc re-executes, clang does not	26
c	clang re-executes, gcc does not	8
d	neither re-executes	100
paired difference	(b - c)/149	+12.1 pt	exact McNemar p = 0.0029
Newcombe paired 95% CI	[4.6, 19.6] pt	excludes 0	bootstrap [4.7, 19.5] pt agrees to < 0.1 pt
15 functions excluded for truncating in either arm (164 - 15 = 149); the 2 gcc-truncated ids {147, 160} are a subset of the 15 clang-truncated ones. Neither arm is obfuscated; both are config 0, the clean grid arm.

The model had not been beaten by obfuscation. It had been shown a dialect it never trained on.

Everything about the semantics of those 149 functions is identical across the arms, same source, same behaviour, same optimization level. What differs is register allocation, instruction selection, idiom choice, prologue shape: the accent of the codegen, and that accent is worth twelve points of measured “resistance” that no obfuscator produced and no paper would have attributed to anything else.

The law underneath¶

The general statement is the one that carries outside this field:

A model evaluated on data generated by a different toolchain than its training distribution is measuring distribution shift, not a security boundary.

The word “toolchain” is doing the general work there. It means whatever pipeline turned the world into your model’s input tokens. If the treatment arm’s data came out of a different pipeline than the training data did, the degradation you measure contains a dialect term, and the dialect term is invisible in the table.

The shape is easy to find elsewhere. An OCR system evaluated on documents from a scanner nobody trained on. A speech model whose “noisy” split was re-encoded through a different codec than the clean one. A model eval whose “hard” subset was regenerated by a newer formatter, so the hard split is also the unfamiliar-formatting split. In each case the measurement is real, reproducible, and mis-labelled.

The test is mechanical. List every pipeline stage that produced your treatment arm; cross out the ones that also produced your training data; whatever is left is inside your number. A treatment arm built by a different tool is the most common way that list ends up non-empty.

Three terms, measured separately¶

The fix follows directly: if the environment carries effects, measure the environment’s effects as their own contrasts. We named three, and measured each one as an isolated paired comparison, each with its own zero base, its own interval, and its own contrast.

Compiler provenance is the one above. The second is disassembler format, the model’s actual input is text that a disassembler printed, and two disassemblers print the same instructions with different syntax and layout conventions. Swapping GNU objdump text for llvm-objdump text is worth +5.6 points at n = 162, p = 0.049. The third is prompt fidelity: whether the prompt tells the model which optimization level it is looking at. That is worth +1.9 points, p = 0.58.

The scoreboard follows, with the two rows that lose kept in.

environment term	contrast	n	test	status
compiler provenance	GCC 27.5% (41/149) vs clang 15.4% (23/149), +12.1pt	149	paired McNemar p = 0.0029; Newcombe [+4.6, +19.6]pt; bootstrap agrees to within 0.1pt	survives
disassembler format	GNU objdump vs llvm-objdump, opt-unaware, +5.6pt	162	p = 0.049	marginal, does not survive Holm
prompt fidelity	opt-aware vs opt-unaware prompt, +1.9pt	162	p = 0.58	demoted to sign-consistent only

Holm is the correction you owe a family of tests: run three of them and the bar for each one rises, because the chance that something clears 0.05 rises with the count. The format term at 0.049 clears an uncorrected threshold by a hair and does not clear a family-wise one. One discordant function flipping the other way erases it. The prompt term’s interval spans zero, and the paper demoted it in public to a sign-consistent observation, which is all an interval spanning zero will support.

Two demoted rows out of three named terms is the honest yield of this leg. They are printed at full size because both of them lose.

data table

term	contrast	n	test	p	Holm threshold	Newcombe 95% CI	status
compiler provenance	+12.1 pt	149	exact McNemar	0.0029	0.0167	[4.6, 19.6] pt	survives
disassembler format	+5.6 pt	162	exact McNemar	0.049	0.025	[0.6, 10.5] pt	fails Holm
prompt fidelity	+1.9 pt	162	exact McNemar	0.5811	0.05	[-2.6, 6.3] pt	demoted (sign-consistent only)
Holm-Bonferroni over the 3-term family, sorted ascending: 0.0167, 0.025, 0.05. Provenance 0.0029 ≤ 0.0167 rejects the null; format 0.049 > 0.025 does not, and everything above it is retained. One discordant flip in the format term takes its p to 0.077 and its difference to +4.9 pt.

The terms do not sum¶

Add the two measured effects to the base cell and the fourth corner should read 25.3 + 5.6 + 1.9 = 32.8%. It reads 27.2%. The prediction misses by about 5.6 points, and in the paper’s own words the three terms “are not additive shares, and do not sum.”

Here is the full 2 × 2 that says so, four obfuscation-free cells, all on GCC binaries, gcc-lenient scorer, native 4096-token window, x86 at O2, n = 162, with Wilson 95% intervals in brackets (the interval for a proportion when the count is small enough that the textbook formula misbehaves):

	opt-unaware prompt	opt-aware prompt
llvm-objdump	25.3% (41/162) [19.2, 32.5]	27.2% (44/162) [20.9, 34.5]
GNU objdump	30.9% (50/162) [24.3, 38.4]	27.2% (44/162) [20.9, 34.5]

Both opt-aware cells come out at the same count, 44 of 162. The format gain that is worth +5.6 points under opt-unaware prompting is worth nothing at all once the prompt names the optimization level, and that is where the missing 5.6 points went.

The 41 in 41/162 (llvm-objdump, opt-unaware) is a different cell from the 41 in 41/149 (the GCC provenance arm). Same count, different populations, different experiments. Anchor a count to its denominator every time you move it between paragraphs, or a number-scrape will eventually splice two of these into a trend.

The format term is +5.6 points and the additivity residual is ~5.6 points, and those are different quantities that happen to coincide. The first is a measured paired difference between two disassembler formats under opt-unaware prompting. The second is arithmetic: predicted corner minus measured corner. And the coincidence is thinner than it looks, because the two 44s are not the same 44 functions. Checked pairwise against the per-function records, the two opt-aware cells share 37 recovered functions and differ on 7 in each direction, so the shortfall equals the vanished term arithmetically rather than because the same functions recovered.

That locates the interaction. It does not explain it. Why conditioning a prompt on optimization level should absorb a disassembler-syntax effect is unexplained here and unexplained in the paper. The four intervals overlap visibly, so the honest label on the interaction is suggestive and unestablished. This section was first drawn as a waterfall, the naive total on the left, three descending steps, one per term, landing on the corner cell, and a waterfall’s whole gestalt is additivity: the reader’s eye completes the sum before it reaches any caption disclaiming one. The arithmetic above forbids that chart, which is why it is not here. In what replaced it the gap is drawn as empty space, because there is no mechanism to fill it with.

data table

cell	prompt	disassembler	k/n	rate	Wilson 95%
A	opt-unaware	llvm-objdump	41/162	25.3%	[19.2, 32.5]
B	opt-unaware	GNU objdump	50/162	30.9%	[24.3, 38.4]
C	opt-aware	llvm-objdump	44/162	27.2%	[20.9, 34.5]
D	opt-aware	GNU objdump	44/162	27.2%	[20.9, 34.5]
predicted corner	25.3 + 5.6 (disassembler) + 1.9 (prompt), assuming the terms add		32.8%, a prediction, never a measured cell; 5.6 pt above the measured 27.2% corner
Cells C and D land on the same 44/162 but are two distinct measured cells; they share 37 recovered func ids and differ on 7 in each direction. All four Wilson intervals overlap and 32.8 sits inside cell D’s [20.9, 34.5], so the non-additivity is suggestive, not established. The gap is left open, not explained.

And there is a second, larger hole just outside this frame. Our matched corner is 27.2% (44/162, our harness). The published cell it corresponds to, clean re-executability at O2 from GNU objdump text, in the source model’s own harness, is 36.71%, with no confidence interval published. That is about 9.5 points we cannot decompose. Two harnesses measuring what is nominally the same quantity disagree by more than any term this piece measured, which sets the scale for how far a single number from one harness can ever be carried toward a number from another.

The same comparison comes out wider at a different aggregation level. Averaged over four optimization levels, O0 through O3, least optimized to most, our reproduction reads 29.51% (O0 43.7 / O1 24.7 / O2 25.3 / O3 24.3) against their 45.37% (68.05 / 39.51 / 36.71 / 37.20), a roughly 16-point miss. Two residuals at two aggregation levels, and they do not belong in the same frame.

We have three candidate causes and no weights for any of them: their exact re-executability scorer, full-object disassembly against our single-symbol disassembly, and their generation stack against ours. Reported open. Also worth stating plainly, since the whole comparison leans on it: identifying their published table’s checkpoint as the release we ran is an inference from matching numbers and matching release dates; the source paper prints no revision string at all.

data table

cell	harness	rate	k/n	CI 95%	disassembler
their published O2 cell	LLM4Decompile End-6.7B, as published	36.71%	not published	none published, upper reference only	GNU objdump
our matched corner	stock v1.5, our harness	27.2%	44/162	[20.9, 34.5] Wilson	GNU objdump
cross-harness gap	36.71 - 27.1605		= 9.5495 = ~9.5 pt, a measured difference of two positions, not decomposed
Candidate causes, unweighted: their exact re-executability scorer; full-object objdump -d vs our single-symbol disassembly; HuggingFace greedy generation vs vLLM. The source paper prints no revision string; identifying their checkpoint as v1.5 is the load-bearing assumption of this comparison.

What this bounds, and what it does not¶

This is one leg, and the leg has walls. One model family at the 6.7B scale, x86 at O2, one re-executability scorer (gcc-lenient), one 4096-token native window, one seed for the factorial cells, and the denominators in the margin note above. Move any of those and every cell has to be recomputed from scratch, the truncation rates in particular are properties of a (window, disassembler, tokenizer) triple, and instances that overflow the window here are excluded from the denominator entirely, which is why 15 functions are missing from the paired 149.

Every recovery number in this piece is re-executability: emitted C that compiles and behaves like the original. Re-executability has its own axis, recovering the shape of a function and recovering a working function are different measurements with different scales, and a structural or control-flow-similarity score from another paper cannot be read across to any cell here.

And the claim is about the instrument: a published deterrence number carries a term nobody named, which is a statement about how the field measures. The attacker in this leg is stock and obfuscation-naive. These numbers bound the instrument, not any shipped product, and a better-resourced attacker only moves them up.

What a controlled comparison has to show¶

what read true	what was actually true	the rule it became
one knob moved, so the metric moved because of the knob	the treatment arm was also a different toolchain: 27.5% vs 15.4% on 149 obfuscation-free functions	list every pipeline stage the treatment arm changed, then measure the ones you did not intend to change
three effects, three numbers, add them up	predicted corner 32.8%, measured 27.2%, ~5.6 points unaccounted	a decomposition owes you its residual as its own printed row, undistributed across the terms
the marginal term at p = 0.049 is a finding	it does not clear a Holm-corrected threshold, and one discordant flip erases it	correct for the family you actually ran, and keep the demoted row at full size
our reproduction matched the published number closely enough	27.2% here against 36.71% published, about 9.5 points, cross-harness	a gap you cannot decompose is a gap you cannot subtract

The null arm turned out to be the largest, best-attested effect in this leg. Two arms, the same functions, the same model, nothing done to either one except handing them to a different compiler, and it moved the metric further than the obfuscation-strength cells it was built to audit, 35 of whose 37 ship as one bare scalar. Run first rather than last, it would have put a provenance term on every obfuscation cell downstream from the beginning. The 12.1 points were sitting there the whole time, in every paper that ran clean-versus-obfuscated across a toolchain boundary, wearing the obfuscator’s name.

The first piece conditioned the metric on whether the transform had changed the function at all; this one decomposed the environment. The next piece changes the attacker, and the number that had survived every correction so far does not survive that one.

Coverage and provenance hide in the same place: a term sits inside the average and nobody prints it. Any eval that folds inputs a transform never touched into the same average, without reporting how often it fired, inherits this exact bug.

Bigger didn't break it. Knowing did.

Thu, 30 Jul 2026 00:00:00 GMT

TL;DR — A neural decompiler, scored gcc-lenient at its native 4096-token window, recovered working code from 18.8% of clean functions and from 0.0-0.9% of the functions each transform actually fired on — its own denominator per cell, never the clean 149. That looked like hardening, so we upgraded the attacker twice. Scaling it from 6.7B to 22B parameters, evaluation held fixed, produced no monotonic gain in the widest-denominator cell — three overlapping points at n = 151. Fine-tuning it on the same transform family recovered 12-16% — from the identical binaries. The leg stayed in the paper; the label changed from “hardening” to “brittleness against a naive attacker.” Preprint: doi:10.5281/zenodo.21682266.

An obfuscation result reads like a property of the binary. Turn the defence on, watch the AI decompiler’s success rate fall to zero, call the code protected.

Then train that same decompiler on the same transforms. It reads 12 to 16% of the very same bytes straight back.

So the zero measured what the attacker had never seen. The headline number describes the attacker at least as much as the defence.

Both halves were measured here on one corpus. A neural decompiler — a transformer that reads stripped assembly, machine instructions with the function and variable names thrown away, and writes back source code, recovered working C from 18.8% of clean functions, and from 0.0% of the 97 functions a constant-encryption pass actually fired on. The comparison was paired, function by function, with a negative control sitting beside it, and every cell in the piece is scored one way throughout: gcc-lenient, at the model’s native 4096-token window.

Measured that way, obfuscation strength is a joint property of two things: what the transform does to the code, and what the attacker happens not to know. Only one of those was in the table.

The cheap objection, that the attacker was simply too small, has a measurable answer, below. The expensive one is that the attacker was never told what had been done to the code. That is the one that landed, and it cost the word hardening in a results table.

The collapse, steelmanned¶

The experiment is easy to state. Take 164 C functions, compile them for x86 at O2 (the optimization setting most shipped software is built with), disassemble them, and ask the model for source that compiles and reproduces the reference behaviour on its test inputs. That strict metric is re-executable recovery, output that merely looks like the original scores zero. On clean binaries this attacker recovers 18.8% of them, 28 of 149.

The 149 is the shared non-truncated subset: 149 of the 164 functions, with 15 dropped because their disassembly did not fit the model’s native 4096-token window. It is the same 149 the previous article’s provenance leg scores on. That article reads a 27.5% clean rate (41/149) there, and the whole of the difference is in the attacker and the toolchain: that arm is the stock released v1.5 checkpoint on gcc-built binaries, this one is a fine-tuned clang-matched control on clang-built ones (its clang-arm counterpart there reads 15.4%). A clean rate is a property of the attacker and the toolchain, so the two never belong on one axis and must never be quoted as a range.

Then run each transform and re-measure, counting only the functions where the transform actually changed the object’s bytes and the prompt still fit the window. That conditioning is fired-only, and the first piece in this series spends its whole length arguing for it: a transform that never touched a function has no business collecting that function’s clean recovery rate on its behalf.

Constant encryption: 0.0% (0 of 97). Control-flow flattening: 0.0% (0 of 63). Bogus control flow: 0.9% (1 of 106). Those cells are the original seed, 0xC0FFEE; two further held-out seeds reproduce them, with flattening firing on 64 functions instead of 63 in one of them. The scorer is the more permissive of the two re-executability scorers we ran, and the window is the model’s native 4096 tokens throughout.

If you read the first piece, you will notice flattening and bogus control flow have swapped places since its raw aggregates. That swap is itself a finding: at these counts neither ordering is real, and which one you get depends on the regime you measured in. That piece’s fired-only cells sit in a 0-1.4% band against a 9.15% clean anchor, measured with the strict clang-side scorer at an 8192-token window, a separate measurement on a separate axis from these.

Those three denominators come from one corpus of 164 functions, and they differ because the transforms differ. A pass that rewrites every function fires on more of them than a pass that rewrites string literals; a pass that inflates the instruction count pushes more of the surviving functions past the window, where they are scored as failures for reasons that have nothing to do with security. Each cell therefore ends at its own n, and putting a count from one beside a rate from another is its own defect.

Against an 18.8% clean base, that is a collapse to roughly nothing, and it holds up under the checks that usually break this kind of result. The underlying effect survives a paired McNemar test, the standard test for a before/after flip measured on the same items, at p = 2.4e-4, with a placebo config as its negative control. That placebo is a backend-marker pass that perturbs 6 of the 164 objects and leaves every per-function outcome identical to clean; that outcome identity was established in the first article’s clang-strict arm, and what the placebo does here is show that the pipeline does not manufacture a drop when nothing meaningful was done to the code.

So: a large drop, on matched functions, with a paired test and a control. Stopping there ships a deterrence claim with real statistics behind it.

Maybe the attacker was just small¶

That is the first objection anyone sensible raises, and prose cannot answer it. A 6.7B-parameter decompiler failing on obfuscated input is compatible with two very different worlds: one where the transforms genuinely destroy recoverable structure, and one where the model was never big enough to read through anything harder than an unobfuscated function. Those worlds are distinguishable only by making the attacker bigger and holding everything else still.

So we ran the same six configurations, clean, string encryption, constant encryption, control-flow flattening, bogus control flow, and the heavy stack of all three, against three obfuscation-naive attackers at 6.7B, 9B and 22B parameters, same evaluation, same seed 0xC0FFEE. All three are stock LLM4Decompile-v2 Ghidra-refinement checkpoints: one published family at three sizes, each reading a traditional decompiler’s pseudo-C rather than raw disassembly. Holding the evaluation fixed is the whole design, and it is the part that gets quietly dropped in scaling comparisons: change the model and the prompt and the window and the scorer together, and any difference you find belongs to the bundle rather than the parameter count. Here only the weights move. The cell to watch is bogus control flow, because it is the obfuscated configuration with the largest number of functions that both fired and survived the window, the widest denominator in the set, so the least noisy place to look for a scaling effect.

It reads 18.5% [13.2, 25.5] at 6.7B, 15.9% (24 of 151) [10.9, 22.6] at 9B, and 19.9% (30 of 151) [14.3, 26.9] at 22B, brackets being Wilson 95% intervals, the interval you get for a proportion when the count is small enough that the textbook formula misbehaves. A 3.3x increase in parameters moves the middle rung down and the top rung barely above the bottom. Every interval overlaps every other interval.

The first minted version of the preprint labelled this three-rung series the “maximum” read-through per scale. That overstates it: at 9B the maximum across configurations is 16.7%, from string encryption on a 24-function fired subset, a different 24 from the 24 of 151 in the bogus-control-flow cell above. The series itself is correct and stays, and so does every digit in it; the word “maximum” is what goes.

The obvious escape is a ceiling effect, perhaps the bigger models were not actually better at this task at all, so of course obfuscated recovery did not move. The clean bases refuse it: 32.5%, 33.8%, 31.8% across the three scales, already flat before any transform enters the picture. Nothing in this ladder rewards parameters, on clean input or obfuscated.

Two numbers in this piece now sit in obvious tension. Section one reads bogus control flow at 0.9%; this section reads it at 18.5% at the same 6.7B parameter count. Both are correct, and each belongs to its own evaluation family. The ladder’s rungs are stock Ghidra-refinement models reading a traditional decompiler’s pseudo-C; the collapse leg’s attacker is the fine-tuned clang-matched control reading raw disassembly of clang-built objects. Two conditions differ at once, the checkpoint and the input representation, which is enough to establish that the two clean bases do not belong on one axis and not enough to say which of the two moves the number: 32.5% here, about 1.7x the 18.8% there, over 151 functions rather than 149. That 1.7x is left undecomposed. The ladder’s three rungs are comparable to each other and to nothing else in this piece, which is precisely why it can answer the scaling objection and no other question.

The bogus-control-flow cell, the widest obfuscated denominator in the set (n = 151), read across an obfuscation-naive Ghidra-refinement family at 6.7B, 9B and 22B, gcc-lenient scorer at the model's native 4096-token window. Recovery reads 18.5% at 6.7B and 19.9% at 22B with a dip between; the clean bases are already level near 32%, so the parameter count buys no resolvable read-through. Wilson 95% intervals; three points with intervals this wide cannot order a curve.

data table

Rung	obfuscated k/n	Rate	Wilson 95%	clean-base k/n	clean Rate	clean Wilson 95%
6.7B	28/151	18.5%	[13.2, 25.5]	49/151	32.5%	[25.5, 40.3]
9B	24/151	15.9%	[10.9, 22.6]	51/151	33.8%	[26.7, 41.6]
22B	30/151	19.9%	[14.3, 26.9]	48/151	31.8%	[24.9, 39.6]
Denominator family: 151 (Ghidra-Ref, obfuscation-naive), a third family that never shares an axis with the End leg’s 149 / 106 / 97. The three obfuscated intervals share the whole range [14.3, 22.6].

The honest caption on that ladder avoids the word “flat.” Three points with intervals that wide are compatible with a rising curve, a falling curve and a level one; a reader whose intuition lands inside them has not been refuted. What the ladder does establish is narrower and sufficient: within this range, on this evaluation, parameter count does not buy read-through.

Scale is not the variable this leg tests.

Tell the attacker what you did¶

Kerckhoffs published the principle in 1883 for ciphers, and it transfers without modification: a system has to stay secure when the adversary knows exactly how it works. Every number in section one was measured against an adversary who did not. The naive attacker had seen ordinary compiled code and had never seen a flattened control-flow graph or an encrypted constant pool in training. Its 0.0% is a measurement of that gap as much as of the transform.

There is a mechanical reason to expect this to matter, and it is the least surprising thing in the article. These transforms are compiler passes. A pass emits the same idioms every time it runs, the same dispatcher shape for a flattened graph, the same decrypt-then-use sequence around a hidden constant, because that is what a pass is. Those idioms are precisely the kind of regularity a sequence model absorbs from examples. Withholding them from the attacker’s training set is a choice about the experiment, and the experiment then reports the consequences of that choice as a property of the binary.

So we removed the gap. Fine-tune the attacker on the same transform family, then re-run the identical evaluation, same binaries, same functions, same window, same scorer, same 18.8% clean baseline. Only the attacker’s training distribution changed.

Constant encryption: 14.4% (14 of 97) [8.8, 22.8]. Control-flow flattening: 14.3-15.6%. Bogus control flow: 12.3% (13 of 106) [7.3, 19.9]. The heavy composite, all transforms stacked: 16.0%.

Read the constant-encryption cell twice. It is 0 of 97 for the naive attacker and 14 of 97 for the obfuscation-aware one. Same denominator, same functions, same bytes on disk. Nothing about the binary changed between those two numbers.

One set of binaries scored by two attackers that differ only in what they were trained on. The obfuscation-naive control reads 0.0-0.9% of the functions each transform actually fired on; the obfuscation-aware Kerckhoffs control reads 12.3-16.0% of the identical bytes, gcc-lenient scorer, native 4096-token window, clang x86/O2, fired-only, each cell on its own denominator. One modest 6.7B fine-tuned attacker on the static client-side tier: a floor, not a ceiling.

data table

Config	cfg	Attacker	Firing count (/164)	Truncated	n	k	Rate	Wilson 95%
string_encryption	cfg 1	naive	29	6	23	0	0.0%	[0.0, 14.3]
string_encryption	cfg 1	aware	29	6	23	1	4.3%	[0.8, 21.0]
constant_encryption	cfg 2	naive	157	60	97	0	0.0%	[0.0, 3.8]
constant_encryption	cfg 2	aware	157	60	97	14	14.4%	[8.8, 22.8]
flatten_cfg	cfg 4	naive	139	76	63	0	0.0%	[0.0, 5.7]
flatten_cfg	cfg 4	aware	139	76	63	9	14.3%	[7.7, 25.0]
bogus_control_flow	cfg 8	naive	164	58	106	1	0.9%	[0.2, 5.2]
bogus_control_flow	cfg 8	aware	164	58	106	13	12.3%	[7.3, 19.9]
heavy composite (ce+cff+bcf)	cfg 14	naive	164	139	25	0	0.0%	[0.0, 13.3]
heavy composite (ce+cff+bcf)	cfg 14	aware	164	139	25	4	16.0%	[6.4, 34.7]
Clean-base rail: 28/149 = 18.8% [13.3, 25.8], the one shared non-truncated subset (164 minus the 15 that truncate at the native-4096 window), never a superset the obfuscated cells are drawn from; each obfuscated cell owns its own firing-and-truncation denominator.

A fine-tuned attacker recovering anything invites the accusation that it memorised the answers, so the robustness checks carry weight here. Three held-out seeds, disjoint from training, all give the same band. The read-through is stable across the last five training checkpoints, so where the run happened to halt does not drive it. The fine-tuning corpus runs to 815,000 records with the evaluation seeds held out. And the tell that settles it: recovery varies by configuration, 12.3% here, 16.0% there, where memorisation of a held-out evaluation would show up as a flat rate that ignores what the transform did.

One thing the band does not do is reach the clean base. 12-16% against 18.8% means the transforms still cost the informed attacker something real. The base is computed over all 149 non-truncated functions and each cell over its own fired subset, which is why this bounds attribution but cannot rank transforms; the intervals on the two anchor cells ([8.8, 22.8] and [7.3, 19.9]) overlap almost entirely in any case. The finding is about attribution: most of what the first measurement credited to obfuscation belonged to the attacker’s training set.

The control that was nearly locked¶

That comparison is worth exactly as much as its control model, and the control nearly broke.

The obfuscation-aware attacker has to be compared against a matched naive one trained on the same schedule, or the whole contrast reduces to “we trained one model longer.” While that control was training, its held-out accuracy across three consecutive evaluation points read 23.5%, then 26.2%, then 22.1%, and I read that shape the way the shape invites: plateaued, stop training, lock the checkpoint. Nobody objected.

Then the test got computed instead of the curve eyeballed. Chi-squared across the three points is 0.69, p is about 0.71, indistinguishable from noise. The sample behind each point was roughly one ninth of what would be needed to detect a real trend of that size. There was no plateau in that data, no trend either, and an irreversible decision was one step away from being made on it.

Reading three overlapping estimates as convergence is the identical error this project spends its whole length documenting in other people’s papers, committed here on a training curve in the same month it was being catalogued in a census of ninety-six.

The held-out trace read 23.5%, 26.2%, 22.1% and invited a plateau reading; dropped among nineteen fixed-rate null draws it is not identifiable. The homogeneity chi-square is 0.69 (p about 0.71) and the eval carries roughly a ninth of the sample a +5-point-per-half-epoch trend would need at 80% power, indistinguishable from noise, which is not the same as the rate being constant. Clang-clean re-executable recovery on the held-out eval, N about 149.

data table

Quantity	Value
real trace, step 500 / 1000 / 2200	35/149 (23.5%) / 39/149 (26.2%) / 33/149 (22.1%)
chi-square of homogeneity, df = 2	0.69
p	0.71
pooled rate	107/447 = 23.9%
the reader’s n	149
required per arm, 80% power vs +5 pt / half-epoch	1,280
ratio	roughly 1/9 (149/1280 = 0.116)
per-point standard error	3.5 pt
minimal detectable difference	~9-10 pt unpaired (~8 pt paired McNemar at within-item correlation ~0.6)
Nineteen null panels are binomial(n = 149, p = 0.239); panel 19 is the real trace. The test supports “indistinguishable from noise at this n”, not “the rate is constant”, and is equally consistent with a rising curve.

We reversed the early stop and retrained. The rule that now decides when a run ends adjudicates on held-out loss, requires a full epoch, and fixes one identical schedule across both arms, and it was adopted mid-flight, as an amendment, which is the sort of thing a paper should disclose about its own pre-registration: the stopping rules you commit to in advance, precisely so you cannot pick them after seeing the numbers.

What changed, and what stayed¶

The relabelling is small to describe. “Hardening” and “deterrence” came out of the headline. “Brittleness against a naive attacker” went in, at the same type size, because a correction set smaller than the claim it replaces is not a correction.

The leg itself stayed in the paper. That is the part people misread as soon as they hear the word relabel: the effect is real, it is paired at p = 2.4e-4, it is placebo-controlled, and it reproduces. The defect lived in the sentence attached to the measurement.

The results-table word hardening ruled through and replaced, at the same type size, by brittleness against a naive attacker, the leg kept, not retracted (paired McNemar p = 2.4e-4, placebo-controlled). The clean-arm floor is what fixes the reading: before any transform this one 6.7B naive control already fails to emit compiling C on 51.2% of clean functions (gcc-lenient, native 4096), so the collapse measures attacker weakness, not a property of the binary.

data table

Clean-arm floor	Value
k/n (clean functions, config 0)	84/164
rate	51.2%
Wilson 95%	[43.6, 58.8]
arm	obfuscation-naive control, clean
seed 0xC0FFEE / 0xBADC0DE / 0x5EED5	84/164 at all three
surviving test	exact McNemar p = 2.4e-4, placebo-controlled; aware band 12-16% vs naive maximum 0.9% (1/106)
The bar is a count (84 of 164), not a per-function map. A different arm and scorer (D1C clang-strict) reads a 76.8% clean floor, do not merge it with the 51.2%.

The scoreboard has to include the rows that argue the other way, so: the middle rung of the Ghidra-refinement scaling ladder went down, 18.5% to 15.9%, which is exactly the shape a clean “bigger is better” story would have to explain away, and nothing here explains it. The preprint’s “maximum” label was wrong, and the corrected maximum of 16.7% on a 24-function fired subset is smaller than the number originally published. The naive-attacker collapse survived a deliberate attempt to demote it out of the paper entirely, which makes the reversal here narrower than the word “reversal” usually promises.

Retrain the attacker and the deterrence number moves; the binary sat still the whole time.

What this does not measure¶

The scope is narrow and stating it narrowly is the point. Everything above is the static client-side tier: bytes shipped to a machine you do not control, read by a model, with no server in the loop that could refuse to answer. It is measured against obfuscation-naive attackers at 6.7B, 9B and 22B, and against one modest 6.7B fine-tuned attacker for the Kerckhoffs leg, all of them on the gcc-lenient scorer at the model’s native 4096-token window, with a fixed prompt and a single-shot budget: no tool use, no iteration, no human in the loop. Every number here is re-executability, emitted C that compiles and reproduces the reference behaviour; none of it sits on the same axis as a structural or control-flow-similarity score, and a shape number from another paper cannot be read across to any of these. A qualitatively stronger adversary is the obvious next experiment, and until it runs, every read-through number here is where an informed attacker starts, and the attacker that produced it is part of the number.

Nor does any of it license the inverse claim. Nothing here says a shipped product is broken, and nothing here says client-side protection is worthless, a 12-16% read-through against an informed attacker leaves most of a binary unrecovered. What it buys, in either direction, is deterrence, and deterrence is the whole of what it buys.

It is also worth being clear about who the target is, because a series that audits ninety-six papers reads like an accusation whether or not it is one. No published cell value in that corpus is disputed here, mine included; every number in every one of those papers may well be exactly what its authors measured. The presentation is what fails, a single scalar, published without the attacker, the window, the firing rate or the denominator that produced it, which no reader can take apart afterwards. One of those scalars was mine.

The statistics are thin in the places where the transforms thinned them. The anchor cells rest on 13 and 14 successes; the composite’s 16.0% is the highest number in the band and the one with the thinnest support behind it. Three rungs do not make a scaling law, and the ladder’s intervals are wide enough to accommodate most people’s priors.

What stays open is most of it. A qualitatively stronger attacker, iterative, tool-assisted, allowed more than one shot, has not been run, and the honest expectation is that the 12-16% band moves up. The second article’s additivity residual is still unexplained and still reported as open. And the census the first article is built on found essentially nobody publishing the conditions that would let a reader check any of this, which means the field’s obfuscation-strength numbers remain, one paper at a time, un-recomputable. That is the next measurement, and until somebody runs it, every number above stands where it was left.

#	CLAIM	AS REPORTED	DENOMINATOR	AS AUDITED	WHAT SURVIVED
1	our own grid’s transform ranking (bogus-control-flow crushes recovery, flattening barely dents it)	raw aggregates CFF 6.1% (10/164) vs BCF 0.6% (1/164), an ordering	/164 raw aggregate, and per-config firing denominators 139 / 164 / 157 / 29 once conditioned	fired-only CFF 1.4% (2/139), BCF 0.6% (1/164), const 0.0% (0/157): a 0-1.4% floor with no ordering supported (no pairwise Fisher p below 0.05; min p = 0.22 fired-only, p = 0.50 full-164)	the DISSOLUTION survived and became the finding: the ranking was a coverage artifact. Nothing was deleted, the aggregate is still real, it just does not mean what it appeared to mean.
2	’hardening’ / ‘deterrence’ from generic IR obfuscation	~100% relative reduction: clean 18.8% (28/149) -> obfuscated max 0.9% (1/106)	gcc-lenient, native-4096, per-config n (23 / 97 / 63 / 106 / 25); the clean base is THE 149, the one shared non-truncated subset, which is the same 149 Article 2’s provenance leg scores on (verified 2026-07-29 by set comparison). One construction, not two.	RELABELLED to ‘brittleness against a naive attacker’: the obfuscation-aware Kerckhoffs attacker reads the same configs at 12.3-16.0% off the same 18.8% base	THE LEG WAS KEPT. The effect is real and paired (McNemar p = 2.4e-4, placebo-controlled). Only the LABEL changed, relabelled, not renumbered.
3	the attacker-scaling series is the ‘maximum’ obfuscated read-through	18.5% -> 15.9% -> 19.9% at 6.7B / 9B / 22B, labelled ‘maximum’ in the Zenodo v1 preprint	/151 in every rung (bogus_control_flow’s fired-and-non-truncated n)	the series is the bogus-control-flow cell, not the maximum. At 9B the true maximum is 16.7% from string_encryption, n = 24	every number. RELABELLED, NOT RENUMBERED: the word ‘maximum’ goes, the series stays.
4	the control model had ‘plateaued at step 1000’ and could be locked	held-out 23.5% -> 26.2% -> 22.1% read as a plateau; training stopped at under 0.4 epoch	N ~ 149 held-out PROMPTS, a coincidental eval size, NOT the corpus 149 of row 2 (the shared matched non-truncated subset). Two different objects that happen to share a number; successes ~ {35, 39, 33}	chi-square(2) = 0.69, p = 0.71, indistinguishable from noise, and equally consistent with a rising curve. Detecting +5 pt per half-epoch at 80% power needs N ~ 1,280, roughly 9x the eval	the decision was REVERSED and the control retrained to at least one full epoch; the pre-registration gained a stopping rule (amended 2026-07-18). The owned mistake became the protocol.
5	Plank C (confidence intervals) is met by ‘12 papers (7 firmly + 5 weakly)‘	C = 12 (7 firmly + 5 weakly) over the 91 full-text papers	/91 full-text papers (tiers FT-self + FT-fan); NOT /96 and NOT /37	C = 12 (6 firmly + 6 weakly). The complete re-verification downgraded bcsd_lessons_kim_tse22’s Plank C to YES-weak: it reports fold standard deviations and per-fold ranges, not intervals or a significance test	the TOTAL (12) and the whole prevalence band. Only the sub-split moved, a one-paper overstatement of the CI-adoption trend.
6	this series’ OWN figure A1-F1: ‘the committed prevalence_master.csv still carries the stale cell and was never fixed’	A1-F1 stated in three places of rendered text, a panel note, the footer and a gen.py docstring, that the master still read 7 firmly + 5 weakly and had never been corrected; data/census.json carried master_is_stale = true and gen.py PINNED it with assert SUB[‘master_is_stale’] is True	/91 full-text papers (tiers FT-self + FT-fan), row 5’s denominator, unchanged. This row corrects a claim about a FILE’S CONTENTS, not a measurement, so it adds no new denominator and none of the counts moved.	FALSE as of 2026-07-29. prevalence_master.csv carries plankC = YES-weak for bcsd_lessons_kim_tse22, and a tier-filtered recount over the 91 full-text rows gives 6 firmly + 6 weakly. The manuscript was corrected first and the data second, so the figure, built inside that window, transcribed the pre-correction state and shipped it as a live present-tense claim with a passing assertion holding it in place.	every number. The sub-split was 6 + 6 before and after and the TOTAL of 12 never moved; row 5 is untouched. What changed is the MECHANISM: A1-F1 now READS the CSV cell at build time instead of restating it, and the assertion is re-pinned so that passing means ‘the CSV agrees with the manuscript’, a regression of the cell now fails the build instead of satisfying it.

| what read true | what was actually true | the rule it became | |, |, |, | | an 18.8% -> 0.0% collapse with a paired test behind it | both numbers were right; the second was measuring what the attacker had never seen | report what the attacker was trained on in the same breath as the drop | | the attacker is too small, so scale will shrink the gap | 3.3x the parameters moved the cell 18.5% -> 15.9% -> 19.9%, every interval overlapping | hold the evaluation fixed and print the intervals before calling three points a trend | | held-out accuracy 23.5 -> 26.2 -> 22.1 has plateaued | chi-squared 0.69, p ~ 0.71, at roughly a ninth of the sample needed to see a trend | compute the test before acting on the shape of three numbers | | a relabel is a retraction | the leg was kept and reproduces; only the claim was renamed | print the correction at the same size as the claim it replaces |

The word in the results table was hardening, and the table was arithmetically correct in every cell. What it hid was that its most impressive number described the attacker’s training set. The 0.0% was not semantic hardening; it was attacker ignorance, and ignorance is the one property of an adversary you cannot ship.

Attacker conditioning was the last term this series pulled out of the same average. Coverage was the first. Any eval that folds inputs a transform never touched into the same average, without reporting how often it fired, inherits this exact bug.