The compiler did it, and my obfuscation took the credit

You build the corpus twice. Once clean, once through the obfuscator. Then you report how far the AI decompiler’s success rate fell.

But the interesting obfuscators are compiler passes, so building that second corpus changes the compiler too. On the corpus below, the compiler switch alone is worth 12.1 points of measured resistance — with no obfuscation anywhere in either arm.

The quantity that moves is re-executable recovery. A neural decompiler reads stripped assembly — machine instructions with the function and variable names thrown away — and writes back C. The metric counts the fraction of functions whose emitted C actually compiles and reproduces the original’s behaviour.

It is the strictest metric in this field, and its drop between clean and obfuscated binaries is what gets published as obfuscation strength. Here it moved 12.1 points on a change nobody had proposed as a treatment.

A controlled comparison isolates the variable you named only if that variable was the only one that moved, and in this field switching obfuscators almost always switches compilers too.

The comparison that looks airtight¶

The field’s standard experiment is a good experiment. Compile a corpus clean. Compile it again through an obfuscator. Run the same decompiler on both. Report the drop. One knob moved, one outcome moved, attribute the delta to the knob. That is what a controlled comparison means, and the instinct behind it is sound, it is how you would measure a drug, a cache policy, or a compiler flag.

It is also, overwhelmingly, what gets published. In the census the first piece in this series walks through, 35 of the 37 papers that report an obfuscation-strength number at all report exactly one bare aggregate scalar for it (37 is the obfuscation-arm subset of 91 full-text-verified papers; those denominators are unpacked in that piece).

The steelman is stronger than it looks, because at the level of author intent it is simply true: the only thing the author touched was the obfuscator. Nobody edited the training set. Nobody swapped the model.

But those obfuscators live inside the compiler, at the intermediate-representation level, which means you obfuscate by building your whole corpus with the toolchain the obfuscator plugs into. That toolchain is usually clang/LLVM. And the model in every cell below, LLM4Decompile v1.5, is trained on GCC output, which is a documented property of its released training corpus rather than an inference from my result. The mechanism has independent attestation too: Decaf, an unrelated system, reports that recompiling its inputs with clang instead of GCC degrades its own GCC-trained reranker, and attributes that to assembly patterns the model was never exposed to. So the clean arm and the obfuscated arm differ by two things, and only one of them is in the paper’s table.

One thing changed, and it was the toolchain¶

data table

measured on func_id 0	gcc	clang	difference
instructions in <func0>	27	33	+6 under clang
.text extent (bytes)	92	116	+24 under clang
object file (bytes)	1512	1352	-160 under clang, sections and padding, not a code-size measure
prompt length (tokens)	388	445	+57; both well under the 4096 window, neither truncated
re-executes (gcc-lenient)	yes	no	the flip this figure exists to show
Excerpt is func0 only, one function’s .text out of the 164-function object, nothing inside it elided; offsets, mnemonics and operands are verbatim from the committed listing. Neither arm is obfuscated.

So we built the corpus twice, once with GCC, once with clang, both x86 at O2 (the optimization setting most shipped software is built with), neither arm touched by any obfuscation pass, ran the stock published decompiler (LLM4Decompile v1.5) over both with the same prompt, the same 4096-token native context window, the same seed and the same gcc-lenient re-executability scorer, and compared the two arms function by function, pair by pair, an arm built to read as a null.

GCC: 27.5% (41/149). Clang: 15.4% (23/149). A difference of +12.1 points in favour of the toolchain the model was trained on, on binaries with zero obfuscation in them. Both arms here are stock v1.5. The attacker piece in this series reports a clean rate of 18.8% on these same 149 functions from a different checkpoint, the fine-tuned clang-matched control; a clean rate is a property of the attacker, so those two belong to different experiments and cannot be read as a range.

Four denominators across this series, and they are not nested the way you would guess. 164 is the grid corpus. 149 is the shared matched non-truncated subset, the functions whose disassembly fit the window in both compiler arms, with 15 dropped for truncating in either one. The attacker piece’s clean base sits on this same 149; the two sets were checked function by function and are identical, so there is one 149 in this series rather than two constructions that happen to collide. 162 is the grid minus 2 truncated instances, and it is the denominator for the factorial cells below. 151 is the scaling ladder’s bogus-control-flow cell in the attacker piece, fired and non-truncated at that arm’s window. It nests inside 164 and inside 162, but 7 of its functions fall outside the 149, so 151 and 149 overlap with neither one containing the other. The 15 dropped here are also a different 15 from the 15 recovered functions in the placebo section of the first piece. Putting a count from one of these on an axis with a rate from another is a defect all by itself.

The pairing is the point, so the test respects it. Paired McNemar, a test that ignores every function the two arms agree on and looks only at the ones where they disagree, gives p = 0.0029. The Newcombe paired 95% confidence interval for the difference is [+4.6, +19.6] points, and a bootstrap, resampling the data 20,000 times and reading the spread, which assumes almost nothing about its shape, lands on the same interval to under 0.1 point. Three routes to one answer, and the assumption-light one agrees with the other two.

data table

arm	k/n	rate	Wilson 95%
gcc binaries	41/149	27.5%	[21.0, 35.2]
clang binaries	23/149	15.4%	[10.5, 22.1]
paired 2×2 cell	meaning	count
a	both arms re-execute	15
b	gcc re-executes, clang does not	26
c	clang re-executes, gcc does not	8
d	neither re-executes	100
paired difference	(b - c)/149	+12.1 pt	exact McNemar p = 0.0029
Newcombe paired 95% CI	[4.6, 19.6] pt	excludes 0	bootstrap [4.7, 19.5] pt agrees to < 0.1 pt
15 functions excluded for truncating in either arm (164 - 15 = 149); the 2 gcc-truncated ids {147, 160} are a subset of the 15 clang-truncated ones. Neither arm is obfuscated; both are config 0, the clean grid arm.

The model had not been beaten by obfuscation. It had been shown a dialect it never trained on.

Everything about the semantics of those 149 functions is identical across the arms, same source, same behaviour, same optimization level. What differs is register allocation, instruction selection, idiom choice, prologue shape: the accent of the codegen, and that accent is worth twelve points of measured “resistance” that no obfuscator produced and no paper would have attributed to anything else.

The law underneath¶

The general statement is the one that carries outside this field:

A model evaluated on data generated by a different toolchain than its training distribution is measuring distribution shift, not a security boundary.

The word “toolchain” is doing the general work there. It means whatever pipeline turned the world into your model’s input tokens. If the treatment arm’s data came out of a different pipeline than the training data did, the degradation you measure contains a dialect term, and the dialect term is invisible in the table.

The shape is easy to find elsewhere. An OCR system evaluated on documents from a scanner nobody trained on. A speech model whose “noisy” split was re-encoded through a different codec than the clean one. A model eval whose “hard” subset was regenerated by a newer formatter, so the hard split is also the unfamiliar-formatting split. In each case the measurement is real, reproducible, and mis-labelled.

The test is mechanical. List every pipeline stage that produced your treatment arm; cross out the ones that also produced your training data; whatever is left is inside your number. A treatment arm built by a different tool is the most common way that list ends up non-empty.

Three terms, measured separately¶

The fix follows directly: if the environment carries effects, measure the environment’s effects as their own contrasts. We named three, and measured each one as an isolated paired comparison, each with its own zero base, its own interval, and its own contrast.

Compiler provenance is the one above. The second is disassembler format, the model’s actual input is text that a disassembler printed, and two disassemblers print the same instructions with different syntax and layout conventions. Swapping GNU objdump text for llvm-objdump text is worth +5.6 points at n = 162, p = 0.049. The third is prompt fidelity: whether the prompt tells the model which optimization level it is looking at. That is worth +1.9 points, p = 0.58.

The scoreboard follows, with the two rows that lose kept in.

Holm is the correction you owe a family of tests: run three of them and the bar for each one rises, because the chance that something clears 0.05 rises with the count. The format term at 0.049 clears an uncorrected threshold by a hair and does not clear a family-wise one. One discordant function flipping the other way erases it. The prompt term’s interval spans zero, and the paper demoted it in public to a sign-consistent observation, which is all an interval spanning zero will support.

Two demoted rows out of three named terms is the honest yield of this leg. They are printed at full size because both of them lose.

data table

term	contrast	n	test	p	Holm threshold	Newcombe 95% CI	status
compiler provenance	+12.1 pt	149	exact McNemar	0.0029	0.0167	[4.6, 19.6] pt	survives
disassembler format	+5.6 pt	162	exact McNemar	0.049	0.025	[0.6, 10.5] pt	fails Holm
prompt fidelity	+1.9 pt	162	exact McNemar	0.5811	0.05	[-2.6, 6.3] pt	demoted (sign-consistent only)
Holm-Bonferroni over the 3-term family, sorted ascending: 0.0167, 0.025, 0.05. Provenance 0.0029 ≤ 0.0167 rejects the null; format 0.049 > 0.025 does not, and everything above it is retained. One discordant flip in the format term takes its p to 0.077 and its difference to +4.9 pt.

The terms do not sum¶

Add the two measured effects to the base cell and the fourth corner should read 25.3 + 5.6 + 1.9 = 32.8%. It reads 27.2%. The prediction misses by about 5.6 points, and in the paper’s own words the three terms “are not additive shares, and do not sum.”

Here is the full 2 × 2 that says so, four obfuscation-free cells, all on GCC binaries, gcc-lenient scorer, native 4096-token window, x86 at O2, n = 162, with Wilson 95% intervals in brackets (the interval for a proportion when the count is small enough that the textbook formula misbehaves):

Both opt-aware cells come out at the same count, 44 of 162. The format gain that is worth +5.6 points under opt-unaware prompting is worth nothing at all once the prompt names the optimization level, and that is where the missing 5.6 points went.

The 41 in 41/162 (llvm-objdump, opt-unaware) is a different cell from the 41 in 41/149 (the GCC provenance arm). Same count, different populations, different experiments. Anchor a count to its denominator every time you move it between paragraphs, or a number-scrape will eventually splice two of these into a trend.

The format term is +5.6 points and the additivity residual is ~5.6 points, and those are different quantities that happen to coincide. The first is a measured paired difference between two disassembler formats under opt-unaware prompting. The second is arithmetic: predicted corner minus measured corner. And the coincidence is thinner than it looks, because the two 44s are not the same 44 functions. Checked pairwise against the per-function records, the two opt-aware cells share 37 recovered functions and differ on 7 in each direction, so the shortfall equals the vanished term arithmetically rather than because the same functions recovered.

That locates the interaction. It does not explain it. Why conditioning a prompt on optimization level should absorb a disassembler-syntax effect is unexplained here and unexplained in the paper. The four intervals overlap visibly, so the honest label on the interaction is suggestive and unestablished. This section was first drawn as a waterfall, the naive total on the left, three descending steps, one per term, landing on the corner cell, and a waterfall’s whole gestalt is additivity: the reader’s eye completes the sum before it reaches any caption disclaiming one. The arithmetic above forbids that chart, which is why it is not here. In what replaced it the gap is drawn as empty space, because there is no mechanism to fill it with.

data table

cell	prompt	disassembler	k/n	rate	Wilson 95%
A	opt-unaware	llvm-objdump	41/162	25.3%	[19.2, 32.5]
B	opt-unaware	GNU objdump	50/162	30.9%	[24.3, 38.4]
C	opt-aware	llvm-objdump	44/162	27.2%	[20.9, 34.5]
D	opt-aware	GNU objdump	44/162	27.2%	[20.9, 34.5]
predicted corner	25.3 + 5.6 (disassembler) + 1.9 (prompt), assuming the terms add		32.8%, a prediction, never a measured cell; 5.6 pt above the measured 27.2% corner
Cells C and D land on the same 44/162 but are two distinct measured cells; they share 37 recovered func ids and differ on 7 in each direction. All four Wilson intervals overlap and 32.8 sits inside cell D’s [20.9, 34.5], so the non-additivity is suggestive, not established. The gap is left open, not explained.

And there is a second, larger hole just outside this frame. Our matched corner is 27.2% (44/162, our harness). The published cell it corresponds to, clean re-executability at O2 from GNU objdump text, in the source model’s own harness, is 36.71%, with no confidence interval published. That is about 9.5 points we cannot decompose. Two harnesses measuring what is nominally the same quantity disagree by more than any term this piece measured, which sets the scale for how far a single number from one harness can ever be carried toward a number from another.

The same comparison comes out wider at a different aggregation level. Averaged over four optimization levels, O0 through O3, least optimized to most, our reproduction reads 29.51% (O0 43.7 / O1 24.7 / O2 25.3 / O3 24.3) against their 45.37% (68.05 / 39.51 / 36.71 / 37.20), a roughly 16-point miss. Two residuals at two aggregation levels, and they do not belong in the same frame.

We have three candidate causes and no weights for any of them: their exact re-executability scorer, full-object disassembly against our single-symbol disassembly, and their generation stack against ours. Reported open. Also worth stating plainly, since the whole comparison leans on it: identifying their published table’s checkpoint as the release we ran is an inference from matching numbers and matching release dates; the source paper prints no revision string at all.

data table

cell	harness	rate	k/n	CI 95%	disassembler
their published O2 cell	LLM4Decompile End-6.7B, as published	36.71%	not published	none published, upper reference only	GNU objdump
our matched corner	stock v1.5, our harness	27.2%	44/162	[20.9, 34.5] Wilson	GNU objdump
cross-harness gap	36.71 - 27.1605		= 9.5495 = ~9.5 pt, a measured difference of two positions, not decomposed
Candidate causes, unweighted: their exact re-executability scorer; full-object objdump -d vs our single-symbol disassembly; HuggingFace greedy generation vs vLLM. The source paper prints no revision string; identifying their checkpoint as v1.5 is the load-bearing assumption of this comparison.

What this bounds, and what it does not¶

This is one leg, and the leg has walls. One model family at the 6.7B scale, x86 at O2, one re-executability scorer (gcc-lenient), one 4096-token native window, one seed for the factorial cells, and the denominators in the margin note above. Move any of those and every cell has to be recomputed from scratch, the truncation rates in particular are properties of a (window, disassembler, tokenizer) triple, and instances that overflow the window here are excluded from the denominator entirely, which is why 15 functions are missing from the paired 149.

Every recovery number in this piece is re-executability: emitted C that compiles and behaves like the original. Re-executability has its own axis, recovering the shape of a function and recovering a working function are different measurements with different scales, and a structural or control-flow-similarity score from another paper cannot be read across to any cell here.

And the claim is about the instrument: a published deterrence number carries a term nobody named, which is a statement about how the field measures. The attacker in this leg is stock and obfuscation-naive. These numbers bound the instrument, not any shipped product, and a better-resourced attacker only moves them up.

What a controlled comparison has to show¶

The null arm turned out to be the largest, best-attested effect in this leg. Two arms, the same functions, the same model, nothing done to either one except handing them to a different compiler, and it moved the metric further than the obfuscation-strength cells it was built to audit, 35 of whose 37 ship as one bare scalar. Run first rather than last, it would have put a provenance term on every obfuscation cell downstream from the beginning. The 12.1 points were sitting there the whole time, in every paper that ran clean-versus-obfuscated across a toolchain boundary, wearing the obfuscator’s name.

The first piece conditioned the metric on whether the transform had changed the function at all; this one decomposed the environment. The next piece changes the attacker, and the number that had survived every correction so far does not survive that one.

Coverage and provenance hide in the same place: a term sits inside the average and nobody prints it. Any eval that folds inputs a transform never touched into the same average, without reporting how often it fired, inherits this exact bug.