tagged · 9 articles
evals
Articles
- 2026
- 2026
What your evals never read
Your eval suite scores the function. It never reads the sentence the model reads before deciding to call that function, and that sentence is the one that can lie, be tampered with after you ship, or be obeyed differently on the very next run. First in a series on what agent evals actually have to measure. - 2026
Bigger didn't break it. Knowing did.
Obfuscation dropped this decompiler's recovery from 18.8% of clean functions (28/149) to 0.0-0.9% of the functions each transform fired on (0/97, 0/63, 1/106) — gcc-lenient scorer, native 4096-token window, paired and placebo-controlled. Tripling its parameters bought no monotonic gain in the largest obfuscated cell: three points at n = 151, intervals overlapping. Fine-tuning it on the same transform family bought 12-16% back, and the headline moved from hardening to brittleness — against one modest 6.7B attacker, where every number is a floor. - 2026
Nobody published the denominator
AI-for-security benchmarks that score models on compiled code can report a number nobody can recompute. Of 37 papers reporting an obfuscation-strength number, exactly 1 conditions it on whether the transform actually fired — and that one is a disassembly paper, which scores nothing on re-executability. Our own do-nothing control passes the metric. - 2026
The compiler did it, and my obfuscation took the credit
Two arms of the same 149 functions, neither arm obfuscated — stock LLM4Decompile v1.5, gcc-lenient scorer, native 4096-token window: GCC-built binaries recover at 27.5%, clang-built at 15.4%, paired McNemar p = 0.0029. The confound is the toolchain, and the law generalizes well past binary analysis. The three-term decomposition built to separate those effects then fails to add up, and the ~5.6-point residual is printed open. - 2026
Green is not evidence
An experiment can validate the wrong quantity against the wrong envelope and pass, a self-test canary can fail identically whether the gate works or not, and an audit command can print nothing and exit 0: three gate designs that read green while checking nothing. - 2026
One earthquake pinned my error bound forever
A seismic stream, a global magnitude bound wrecked by a single mainshock, a windowed bound that heals — and a scoreboard where the headline empirical legs of two papers report zero violations. - 2026
The error analysis everyone cites is for a kernel nobody runs
SCAMP and STUMPY accumulate a mean-centred covariance, not the textbook inner product — and the centred case is structurally different floating-point mathematics, not a special case at mu = 0. - 2026
Three axioms and one disclosed seam
Four verification tools, one certificate: every machine-checked theorem mechanically reduced to three classical axioms with zero unfinished proofs — and the one place two provers cannot talk to each other, named right in the theorem statements.