Explainers
Direct answers to the questions people actually arrive with — one question per page, each one answered from work that shipped, with the artifact you can check it against.
Agent-systems reliability
Multi-agent systems that poll, hang, or report finished while work is still in flight.
- How do I get AI coding agents from different vendors to coordinate on one machine without polling?
Give them a shared local event bus and have each agent block on a subscription instead of re-reading a file. The coordination primitive has to live outside every vendor's runtime, because no vendor's runtime can see another's.
- How do I know when a cascade of delegating agents has actually finished?
Not with a timeout. Quiet is not the same as done: an agent that has delegated and is waiting looks identical to one that has stopped. You need a termination-detection protocol that accounts for work in flight, which distributed systems solved decades ago.
I consult on multi-agent systems that poll, hang, or report finished while work is still in flight — how that works →
Eval and measurement validity
Benchmark and eval numbers that cannot be recomputed from anything published.
- Can someone independently recompute a published benchmark number and tell me whether it holds?
Often, and finding out which is most of the work. A recompute takes one number, re-runs it from the published artifacts alone, and returns one of three verdicts: it reproduces, it reproduces at a different value, or it cannot be recomputed and here is the artifact that was missing.
- How do I check whether a published benchmark number measures what it claims to?
Find the denominator. A benchmark number is an average over a population, and most published ones do not report which items were in it, which were skipped, or which were scored as failures for reasons unrelated to the thing being measured.
- What should an agent eval test that my unit tests never touch?
The text the model actually reads. Your unit tests exercise the function body; the agent never sees it. It decides which tool to call from the tool's name, description and parameter docs, and that surface is untested by every suite that calls the function directly.
I consult on benchmark and eval numbers that cannot be recomputed from anything published — how that works →
Uncertainty and numerical verification
Intervals and numerical kernels that are wrong in ways a passing test suite cannot see.
- Why does my FP8 training run diverge when the same model trains fine in bf16?
Bisect the precision, not the hyperparameters. Cast one tensor class back to bf16 at a time, under a fixed seed, until the divergence stops; the class that fixes it names the failure. Lowering the learning rate until it trains is not a fix, it is a measurement of how much precision you lost.
- How do I bootstrap a time series without destroying the dependence in it?
Resample structures that carry the dependence, not individual observations: contiguous blocks, model residuals, a fitted sieve, or wild multipliers. Which one is right depends on which assumption your data actually breaks, and no single method is best across all of them.
- Who can prove a machine-checked floating-point error bound for my numerical kernel?
You state the kernel's worst-case rounding error as a theorem about real arithmetic, prove it in a proof assistant against a formal model of IEEE-754, and check the proof against your real data. Lean, Coq/Flocq and Gappa each do part of that job.
I consult on intervals and numerical kernels that are wrong in ways a passing test suite cannot see — how that works →