Bigger didn't break it. Knowing did.
Contents
TL;DR — A neural decompiler, scored gcc-lenient at its native 4096-token window, recovered working code from 18.8% of clean functions and from 0.0-0.9% of the functions each transform actually fired on — its own denominator per cell, never the clean 149. That looked like hardening, so we upgraded the attacker twice. Scaling it from 6.7B to 22B parameters, evaluation held fixed, produced no monotonic gain in the widest-denominator cell — three overlapping points at n = 151. Fine-tuning it on the same transform family recovered 12-16% — from the identical binaries. The leg stayed in the paper; the label changed from “hardening” to “brittleness against a naive attacker.” Preprint: doi:10.5281/zenodo.21682266.
An obfuscation result reads like a property of the binary. Turn the defence on, watch the AI decompiler’s success rate fall to zero, call the code protected.
Then train that same decompiler on the same transforms. It reads 12 to 16% of the very same bytes straight back.
So the zero measured what the attacker had never seen. The headline number describes the attacker at least as much as the defence.
Both halves were measured here on one corpus. A neural decompiler — a transformer that reads stripped assembly, machine instructions with the function and variable names thrown away, and writes back source code, recovered working C from 18.8% of clean functions, and from 0.0% of the 97 functions a constant-encryption pass actually fired on. The comparison was paired, function by function, with a negative control sitting beside it, and every cell in the piece is scored one way throughout: gcc-lenient, at the model’s native 4096-token window.
Measured that way, obfuscation strength is a joint property of two things: what the transform does to the code, and what the attacker happens not to know. Only one of those was in the table.
The cheap objection, that the attacker was simply too small, has a measurable answer, below. The expensive one is that the attacker was never told what had been done to the code. That is the one that landed, and it cost the word hardening in a results table.
The collapse, steelmanned¶
The experiment is easy to state. Take 164 C functions, compile them for x86 at O2 (the optimization setting most shipped software is built with), disassemble them, and ask the model for source that compiles and reproduces the reference behaviour on its test inputs. That strict metric is re-executable recovery, output that merely looks like the original scores zero. On clean binaries this attacker recovers 18.8% of them, 28 of 149.
The 149 is the shared non-truncated subset: 149 of the 164 functions, with 15 dropped because their disassembly did not fit the model’s native 4096-token window. It is the same 149 the previous article’s provenance leg scores on. That article reads a 27.5% clean rate (41/149) there, and the whole of the difference is in the attacker and the toolchain: that arm is the stock released v1.5 checkpoint on gcc-built binaries, this one is a fine-tuned clang-matched control on clang-built ones (its clang-arm counterpart there reads 15.4%). A clean rate is a property of the attacker and the toolchain, so the two never belong on one axis and must never be quoted as a range.Then run each transform and re-measure, counting only the functions where the transform actually changed the object’s bytes and the prompt still fit the window. That conditioning is fired-only, and the first piece in this series spends its whole length arguing for it: a transform that never touched a function has no business collecting that function’s clean recovery rate on its behalf.
Constant encryption: 0.0% (0 of 97). Control-flow flattening: 0.0% (0 of 63). Bogus control flow: 0.9% (1 of 106). Those cells are the original seed, 0xC0FFEE; two further held-out seeds reproduce them, with flattening firing on 64 functions instead of 63 in one of them. The scorer is the more permissive of the two re-executability scorers we ran, and the window is the model’s native 4096 tokens throughout.
If you read the first piece, you will notice flattening and bogus control flow have swapped places since its raw aggregates. That swap is itself a finding: at these counts neither ordering is real, and which one you get depends on the regime you measured in. That piece’s fired-only cells sit in a 0-1.4% band against a 9.15% clean anchor, measured with the strict clang-side scorer at an 8192-token window, a separate measurement on a separate axis from these.
Those three denominators come from one corpus of 164 functions, and they differ because the transforms differ. A pass that rewrites every function fires on more of them than a pass that rewrites string literals; a pass that inflates the instruction count pushes more of the surviving functions past the window, where they are scored as failures for reasons that have nothing to do with security. Each cell therefore ends at its own n, and putting a count from one beside a rate from another is its own defect.
Against an 18.8% clean base, that is a collapse to roughly nothing, and it holds up under the checks that usually break this kind of result. The underlying effect survives a paired McNemar test, the standard test for a before/after flip measured on the same items, at p = 2.4e-4, with a placebo config as its negative control. That placebo is a backend-marker pass that perturbs 6 of the 164 objects and leaves every per-function outcome identical to clean; that outcome identity was established in the first article’s clang-strict arm, and what the placebo does here is show that the pipeline does not manufacture a drop when nothing meaningful was done to the code.
So: a large drop, on matched functions, with a paired test and a control. Stopping there ships a deterrence claim with real statistics behind it.
Maybe the attacker was just small¶
That is the first objection anyone sensible raises, and prose cannot answer it. A 6.7B-parameter decompiler failing on obfuscated input is compatible with two very different worlds: one where the transforms genuinely destroy recoverable structure, and one where the model was never big enough to read through anything harder than an unobfuscated function. Those worlds are distinguishable only by making the attacker bigger and holding everything else still.
So we ran the same six configurations, clean, string encryption, constant encryption, control-flow flattening, bogus control flow, and the heavy stack of all three, against three obfuscation-naive attackers at 6.7B, 9B and 22B parameters, same evaluation, same seed 0xC0FFEE. All three are stock LLM4Decompile-v2 Ghidra-refinement checkpoints: one published family at three sizes, each reading a traditional decompiler’s pseudo-C rather than raw disassembly. Holding the evaluation fixed is the whole design, and it is the part that gets quietly dropped in scaling comparisons: change the model and the prompt and the window and the scorer together, and any difference you find belongs to the bundle rather than the parameter count. Here only the weights move. The cell to watch is bogus control flow, because it is the obfuscated configuration with the largest number of functions that both fired and survived the window, the widest denominator in the set, so the least noisy place to look for a scaling effect.
It reads 18.5% [13.2, 25.5] at 6.7B, 15.9% (24 of 151) [10.9, 22.6] at 9B, and 19.9% (30 of 151) [14.3, 26.9] at 22B, brackets being Wilson 95% intervals, the interval you get for a proportion when the count is small enough that the textbook formula misbehaves. A 3.3x increase in parameters moves the middle rung down and the top rung barely above the bottom. Every interval overlaps every other interval.
The first minted version of the preprint labelled this three-rung series the “maximum” read-through per scale. That overstates it: at 9B the maximum across configurations is 16.7%, from string encryption on a 24-function fired subset, a different 24 from the 24 of 151 in the bogus-control-flow cell above. The series itself is correct and stays, and so does every digit in it; the word “maximum” is what goes.The obvious escape is a ceiling effect, perhaps the bigger models were not actually better at this task at all, so of course obfuscated recovery did not move. The clean bases refuse it: 32.5%, 33.8%, 31.8% across the three scales, already flat before any transform enters the picture. Nothing in this ladder rewards parameters, on clean input or obfuscated.
Two numbers in this piece now sit in obvious tension. Section one reads bogus control flow at 0.9%; this section reads it at 18.5% at the same 6.7B parameter count. Both are correct, and each belongs to its own evaluation family. The ladder’s rungs are stock Ghidra-refinement models reading a traditional decompiler’s pseudo-C; the collapse leg’s attacker is the fine-tuned clang-matched control reading raw disassembly of clang-built objects. Two conditions differ at once, the checkpoint and the input representation, which is enough to establish that the two clean bases do not belong on one axis and not enough to say which of the two moves the number: 32.5% here, about 1.7x the 18.8% there, over 151 functions rather than 149. That 1.7x is left undecomposed. The ladder’s three rungs are comparable to each other and to nothing else in this piece, which is precisely why it can answer the scaling objection and no other question.
data table
| Rung | obfuscated k/n | Rate | Wilson 95% | clean-base k/n | clean Rate | clean Wilson 95% |
|---|---|---|---|---|---|---|
| 6.7B | 28/151 | 18.5% | [13.2, 25.5] | 49/151 | 32.5% | [25.5, 40.3] |
| 9B | 24/151 | 15.9% | [10.9, 22.6] | 51/151 | 33.8% | [26.7, 41.6] |
| 22B | 30/151 | 19.9% | [14.3, 26.9] | 48/151 | 31.8% | [24.9, 39.6] |
| Denominator family: 151 (Ghidra-Ref, obfuscation-naive), a third family that never shares an axis with the End leg’s 149 / 106 / 97. The three obfuscated intervals share the whole range [14.3, 22.6]. | ||||||
The honest caption on that ladder avoids the word “flat.” Three points with intervals that wide are compatible with a rising curve, a falling curve and a level one; a reader whose intuition lands inside them has not been refuted. What the ladder does establish is narrower and sufficient: within this range, on this evaluation, parameter count does not buy read-through.
Scale is not the variable this leg tests.
Tell the attacker what you did¶
Kerckhoffs published the principle in 1883 for ciphers, and it transfers without modification: a system has to stay secure when the adversary knows exactly how it works. Every number in section one was measured against an adversary who did not. The naive attacker had seen ordinary compiled code and had never seen a flattened control-flow graph or an encrypted constant pool in training. Its 0.0% is a measurement of that gap as much as of the transform.
There is a mechanical reason to expect this to matter, and it is the least surprising thing in the article. These transforms are compiler passes. A pass emits the same idioms every time it runs, the same dispatcher shape for a flattened graph, the same decrypt-then-use sequence around a hidden constant, because that is what a pass is. Those idioms are precisely the kind of regularity a sequence model absorbs from examples. Withholding them from the attacker’s training set is a choice about the experiment, and the experiment then reports the consequences of that choice as a property of the binary.
So we removed the gap. Fine-tune the attacker on the same transform family, then re-run the identical evaluation, same binaries, same functions, same window, same scorer, same 18.8% clean baseline. Only the attacker’s training distribution changed.
Constant encryption: 14.4% (14 of 97) [8.8, 22.8]. Control-flow flattening: 14.3-15.6%. Bogus control flow: 12.3% (13 of 106) [7.3, 19.9]. The heavy composite, all transforms stacked: 16.0%.
Read the constant-encryption cell twice. It is 0 of 97 for the naive attacker and 14 of 97 for the obfuscation-aware one. Same denominator, same functions, same bytes on disk. Nothing about the binary changed between those two numbers.
data table
| Config | cfg | Attacker | Firing count (/164) | Truncated | n | k | Rate | Wilson 95% |
|---|---|---|---|---|---|---|---|---|
| string_encryption | cfg 1 | naive | 29 | 6 | 23 | 0 | 0.0% | [0.0, 14.3] |
| string_encryption | cfg 1 | aware | 29 | 6 | 23 | 1 | 4.3% | [0.8, 21.0] |
| constant_encryption | cfg 2 | naive | 157 | 60 | 97 | 0 | 0.0% | [0.0, 3.8] |
| constant_encryption | cfg 2 | aware | 157 | 60 | 97 | 14 | 14.4% | [8.8, 22.8] |
| flatten_cfg | cfg 4 | naive | 139 | 76 | 63 | 0 | 0.0% | [0.0, 5.7] |
| flatten_cfg | cfg 4 | aware | 139 | 76 | 63 | 9 | 14.3% | [7.7, 25.0] |
| bogus_control_flow | cfg 8 | naive | 164 | 58 | 106 | 1 | 0.9% | [0.2, 5.2] |
| bogus_control_flow | cfg 8 | aware | 164 | 58 | 106 | 13 | 12.3% | [7.3, 19.9] |
| heavy composite (ce+cff+bcf) | cfg 14 | naive | 164 | 139 | 25 | 0 | 0.0% | [0.0, 13.3] |
| heavy composite (ce+cff+bcf) | cfg 14 | aware | 164 | 139 | 25 | 4 | 16.0% | [6.4, 34.7] |
| Clean-base rail: 28/149 = 18.8% [13.3, 25.8], the one shared non-truncated subset (164 minus the 15 that truncate at the native-4096 window), never a superset the obfuscated cells are drawn from; each obfuscated cell owns its own firing-and-truncation denominator. | ||||||||
A fine-tuned attacker recovering anything invites the accusation that it memorised the answers, so the robustness checks carry weight here. Three held-out seeds, disjoint from training, all give the same band. The read-through is stable across the last five training checkpoints, so where the run happened to halt does not drive it. The fine-tuning corpus runs to 815,000 records with the evaluation seeds held out. And the tell that settles it: recovery varies by configuration, 12.3% here, 16.0% there, where memorisation of a held-out evaluation would show up as a flat rate that ignores what the transform did.
One thing the band does not do is reach the clean base. 12-16% against 18.8% means the transforms still cost the informed attacker something real. The base is computed over all 149 non-truncated functions and each cell over its own fired subset, which is why this bounds attribution but cannot rank transforms; the intervals on the two anchor cells ([8.8, 22.8] and [7.3, 19.9]) overlap almost entirely in any case. The finding is about attribution: most of what the first measurement credited to obfuscation belonged to the attacker’s training set.
The control that was nearly locked¶
That comparison is worth exactly as much as its control model, and the control nearly broke.
The obfuscation-aware attacker has to be compared against a matched naive one trained on the same schedule, or the whole contrast reduces to “we trained one model longer.” While that control was training, its held-out accuracy across three consecutive evaluation points read 23.5%, then 26.2%, then 22.1%, and I read that shape the way the shape invites: plateaued, stop training, lock the checkpoint. Nobody objected.
Then the test got computed instead of the curve eyeballed. Chi-squared across the three points is 0.69, p is about 0.71, indistinguishable from noise. The sample behind each point was roughly one ninth of what would be needed to detect a real trend of that size. There was no plateau in that data, no trend either, and an irreversible decision was one step away from being made on it.
Reading three overlapping estimates as convergence is the identical error this project spends its whole length documenting in other people’s papers, committed here on a training curve in the same month it was being catalogued in a census of ninety-six.
data table
| Quantity | Value |
|---|---|
| real trace, step 500 / 1000 / 2200 | 35/149 (23.5%) / 39/149 (26.2%) / 33/149 (22.1%) |
| chi-square of homogeneity, df = 2 | 0.69 |
| p | 0.71 |
| pooled rate | 107/447 = 23.9% |
| the reader’s n | 149 |
| required per arm, 80% power vs +5 pt / half-epoch | 1,280 |
| ratio | roughly 1/9 (149/1280 = 0.116) |
| per-point standard error | 3.5 pt |
| minimal detectable difference | ~9-10 pt unpaired (~8 pt paired McNemar at within-item correlation ~0.6) |
| Nineteen null panels are binomial(n = 149, p = 0.239); panel 19 is the real trace. The test supports “indistinguishable from noise at this n”, not “the rate is constant”, and is equally consistent with a rising curve. | |
We reversed the early stop and retrained. The rule that now decides when a run ends adjudicates on held-out loss, requires a full epoch, and fixes one identical schedule across both arms, and it was adopted mid-flight, as an amendment, which is the sort of thing a paper should disclose about its own pre-registration: the stopping rules you commit to in advance, precisely so you cannot pick them after seeing the numbers.
What changed, and what stayed¶
The relabelling is small to describe. “Hardening” and “deterrence” came out of the headline. “Brittleness against a naive attacker” went in, at the same type size, because a correction set smaller than the claim it replaces is not a correction.
The leg itself stayed in the paper. That is the part people misread as soon as they hear the word relabel: the effect is real, it is paired at p = 2.4e-4, it is placebo-controlled, and it reproduces. The defect lived in the sentence attached to the measurement.
data table
| Clean-arm floor | Value |
|---|---|
| k/n (clean functions, config 0) | 84/164 |
| rate | 51.2% |
| Wilson 95% | [43.6, 58.8] |
| arm | obfuscation-naive control, clean |
| seed 0xC0FFEE / 0xBADC0DE / 0x5EED5 | 84/164 at all three |
| surviving test | exact McNemar p = 2.4e-4, placebo-controlled; aware band 12-16% vs naive maximum 0.9% (1/106) |
| The bar is a count (84 of 164), not a per-function map. A different arm and scorer (D1C clang-strict) reads a 76.8% clean floor, do not merge it with the 51.2%. | |
The scoreboard has to include the rows that argue the other way, so: the middle rung of the Ghidra-refinement scaling ladder went down, 18.5% to 15.9%, which is exactly the shape a clean “bigger is better” story would have to explain away, and nothing here explains it. The preprint’s “maximum” label was wrong, and the corrected maximum of 16.7% on a 24-function fired subset is smaller than the number originally published. The naive-attacker collapse survived a deliberate attempt to demote it out of the paper entirely, which makes the reversal here narrower than the word “reversal” usually promises.
Retrain the attacker and the deterrence number moves; the binary sat still the whole time.
What this does not measure¶
The scope is narrow and stating it narrowly is the point. Everything above is the static client-side tier: bytes shipped to a machine you do not control, read by a model, with no server in the loop that could refuse to answer. It is measured against obfuscation-naive attackers at 6.7B, 9B and 22B, and against one modest 6.7B fine-tuned attacker for the Kerckhoffs leg, all of them on the gcc-lenient scorer at the model’s native 4096-token window, with a fixed prompt and a single-shot budget: no tool use, no iteration, no human in the loop. Every number here is re-executability, emitted C that compiles and reproduces the reference behaviour; none of it sits on the same axis as a structural or control-flow-similarity score, and a shape number from another paper cannot be read across to any of these. A qualitatively stronger adversary is the obvious next experiment, and until it runs, every read-through number here is where an informed attacker starts, and the attacker that produced it is part of the number.
Nor does any of it license the inverse claim. Nothing here says a shipped product is broken, and nothing here says client-side protection is worthless, a 12-16% read-through against an informed attacker leaves most of a binary unrecovered. What it buys, in either direction, is deterrence, and deterrence is the whole of what it buys.
It is also worth being clear about who the target is, because a series that audits ninety-six papers reads like an accusation whether or not it is one. No published cell value in that corpus is disputed here, mine included; every number in every one of those papers may well be exactly what its authors measured. The presentation is what fails, a single scalar, published without the attacker, the window, the firing rate or the denominator that produced it, which no reader can take apart afterwards. One of those scalars was mine.
The statistics are thin in the places where the transforms thinned them. The anchor cells rest on 13 and 14 successes; the composite’s 16.0% is the highest number in the band and the one with the thinnest support behind it. Three rungs do not make a scaling law, and the ladder’s intervals are wide enough to accommodate most people’s priors.
What stays open is most of it. A qualitatively stronger attacker, iterative, tool-assisted, allowed more than one shot, has not been run, and the honest expectation is that the 12-16% band moves up. The second article’s additivity residual is still unexplained and still reported as open. And the census the first article is built on found essentially nobody publishing the conditions that would let a reader check any of this, which means the field’s obfuscation-strength numbers remain, one paper at a time, un-recomputable. That is the next measurement, and until somebody runs it, every number above stands where it was left.
| # | CLAIM | AS REPORTED | DENOMINATOR | AS AUDITED | WHAT SURVIVED |
|---|---|---|---|---|---|
| 1 | our own grid’s transform ranking (bogus-control-flow crushes recovery, flattening barely dents it) | raw aggregates CFF 6.1% (10/164) vs BCF 0.6% (1/164), an ordering | /164 raw aggregate, and per-config firing denominators 139 / 164 / 157 / 29 once conditioned | fired-only CFF 1.4% (2/139), BCF 0.6% (1/164), const 0.0% (0/157): a 0-1.4% floor with no ordering supported (no pairwise Fisher p below 0.05; min p = 0.22 fired-only, p = 0.50 full-164) | the DISSOLUTION survived and became the finding: the ranking was a coverage artifact. Nothing was deleted, the aggregate is still real, it just does not mean what it appeared to mean. |
| 2 | ’hardening’ / ‘deterrence’ from generic IR obfuscation | ~100% relative reduction: clean 18.8% (28/149) -> obfuscated max 0.9% (1/106) | gcc-lenient, native-4096, per-config n (23 / 97 / 63 / 106 / 25); the clean base is THE 149, the one shared non-truncated subset, which is the same 149 Article 2’s provenance leg scores on (verified 2026-07-29 by set comparison). One construction, not two. | RELABELLED to ‘brittleness against a naive attacker’: the obfuscation-aware Kerckhoffs attacker reads the same configs at 12.3-16.0% off the same 18.8% base | THE LEG WAS KEPT. The effect is real and paired (McNemar p = 2.4e-4, placebo-controlled). Only the LABEL changed, relabelled, not renumbered. |
| 3 | the attacker-scaling series is the ‘maximum’ obfuscated read-through | 18.5% -> 15.9% -> 19.9% at 6.7B / 9B / 22B, labelled ‘maximum’ in the Zenodo v1 preprint | /151 in every rung (bogus_control_flow’s fired-and-non-truncated n) | the series is the bogus-control-flow cell, not the maximum. At 9B the true maximum is 16.7% from string_encryption, n = 24 | every number. RELABELLED, NOT RENUMBERED: the word ‘maximum’ goes, the series stays. |
| 4 | the control model had ‘plateaued at step 1000’ and could be locked | held-out 23.5% -> 26.2% -> 22.1% read as a plateau; training stopped at under 0.4 epoch | N ~ 149 held-out PROMPTS, a coincidental eval size, NOT the corpus 149 of row 2 (the shared matched non-truncated subset). Two different objects that happen to share a number; successes ~ {35, 39, 33} | chi-square(2) = 0.69, p = 0.71, indistinguishable from noise, and equally consistent with a rising curve. Detecting +5 pt per half-epoch at 80% power needs N ~ 1,280, roughly 9x the eval | the decision was REVERSED and the control retrained to at least one full epoch; the pre-registration gained a stopping rule (amended 2026-07-18). The owned mistake became the protocol. |
| 5 | Plank C (confidence intervals) is met by ‘12 papers (7 firmly + 5 weakly)‘ | C = 12 (7 firmly + 5 weakly) over the 91 full-text papers | /91 full-text papers (tiers FT-self + FT-fan); NOT /96 and NOT /37 | C = 12 (6 firmly + 6 weakly). The complete re-verification downgraded bcsd_lessons_kim_tse22’s Plank C to YES-weak: it reports fold standard deviations and per-fold ranges, not intervals or a significance test | the TOTAL (12) and the whole prevalence band. Only the sub-split moved, a one-paper overstatement of the CI-adoption trend. |
| 6 | this series’ OWN figure A1-F1: ‘the committed prevalence_master.csv still carries the stale cell and was never fixed’ | A1-F1 stated in three places of rendered text, a panel note, the footer and a gen.py docstring, that the master still read 7 firmly + 5 weakly and had never been corrected; data/census.json carried master_is_stale = true and gen.py PINNED it with assert SUB[‘master_is_stale’] is True | /91 full-text papers (tiers FT-self + FT-fan), row 5’s denominator, unchanged. This row corrects a claim about a FILE’S CONTENTS, not a measurement, so it adds no new denominator and none of the counts moved. | FALSE as of 2026-07-29. prevalence_master.csv carries plankC = YES-weak for bcsd_lessons_kim_tse22, and a tier-filtered recount over the 91 full-text rows gives 6 firmly + 6 weakly. The manuscript was corrected first and the data second, so the figure, built inside that window, transcribed the pre-correction state and shipped it as a live present-tense claim with a passing assertion holding it in place. | every number. The sub-split was 6 + 6 before and after and the TOTAL of 12 never moved; row 5 is untouched. What changed is the MECHANISM: A1-F1 now READS the CSV cell at build time instead of restating it, and the assertion is re-pinned so that passing means ‘the CSV agrees with the manuscript’, a regression of the cell now fails the build instead of satisfying it. |
| what read true | what was actually true | the rule it became | |, |, |, | | an 18.8% -> 0.0% collapse with a paired test behind it | both numbers were right; the second was measuring what the attacker had never seen | report what the attacker was trained on in the same breath as the drop | | the attacker is too small, so scale will shrink the gap | 3.3x the parameters moved the cell 18.5% -> 15.9% -> 19.9%, every interval overlapping | hold the evaluation fixed and print the intervals before calling three points a trend | | held-out accuracy 23.5 -> 26.2 -> 22.1 has plateaued | chi-squared 0.69, p ~ 0.71, at roughly a ninth of the sample needed to see a trend | compute the test before acting on the shape of three numbers | | a relabel is a retraction | the leg was kept and reproduces; only the claim was renamed | print the correction at the same size as the claim it replaces |
The word in the results table was hardening, and the table was arithmetically correct in every cell. What it hid was that its most impressive number described the attacker’s training set. The 0.0% was not semantic hardening; it was attacker ignorance, and ignorance is the one property of an adversary you cannot ship.
Attacker conditioning was the last term this series pulled out of the same average. Coverage was the first. Any eval that folds inputs a transform never touched into the same average, without reporting how often it fired, inherits this exact bug.