Why a machine must be allowed to be wrong

I built meta-science for the All Things Agentic Hackathon, and I wrote this piece for the purposes of entering that hackathon. Code, demo and video are linked at the end — all of it GPL-3.0.

The claim that cannot fail

"Self-improving AI" is the most repeated and least examined claim in our field. Its problem is not that it is false; its problem is that, as usually stated, it cannot be false.

A system that reports its own improvement is indistinguishable from a system that logs the word "improved" and changes nothing. Demos show successes, benchmarks are chosen after the fact, and the reader is asked to extend trust precisely where trust has no purchase. Karl Popper named this failure a century ago: a theory that cannot be refuted by any conceivable event is not thereby strong — it is thereby empty.

So I did not set out to build a self-improving agent. I set out to build the gate such an agent would have to pass — and only then the agent.

Two axioms

Everything descends from two refusals, stated up front.

No agent is the judge of its own claims — A ⊢ c ⇏ ⊨ c: that an agent derives a claim never entails that the claim holds. The model proposes hypotheses, designs experiments, and suggests improvements to its own method. It never scores itself, never sees the held-out worlds it will be judged on, and never writes to canon. Verdicts are computed, not solicited. This is not distrust of any particular model; it is the constitutional insight behind separated powers and referees who do not play.

Seeing is not doing. Pearl's distinction — conditioning is not intervening, P(Y | do(X)) ≢ P(Y | X) — is what makes discovery cost something. Not-identical rather than never-equal: the two coincide exactly when confounding is absent, and that absence is what may never be assumed. My worlds contain traps where the data, read passively, is clean, strong, and points the wrong way. Reading observations alone, the model recovers the causal direction in zero of four confounded worlds; the same loop, allowed to intervene, recovers four of four. Knowledge that can be had by looking is retrieval. Knowledge that must be paid for in experiments is science.

The physicist's objection

A physicist will object to the second axiom, and did, in the person of this project's author: quantum mechanics knows no passive spectator — observation is interaction. The objection sharpened us twice.

First, the axiom's substance survives it: even in the quantum formalism, conditioning on an outcome (post-selection) and preparing a state (intervention) remain different operations; the slogan frays, the mathematics holds.

Second, pushed further, the objection revealed what a simulated benchmark is: the one place where the observer genuinely stands outside the ontology — because we built the ontology and enumerated its every edge. The side effects are real, and they live in the
host universe, not the guest one. Our determinism tests double as the experimental proof: the same seed builds bit-identical worlds on a hot CPU or a cold one. And where our own API leaks a miniature observer effect — unseeded observation advances the random
stream — we name the crack rather than polish it.

What a refusal is worth

The agent's proposals land in a tier that is non-authoritative by construction.

Promotion to canon requires beating the incumbent on worlds the proposer cannot see or enumerate, by a margin — because a gate without a margin ratchets on noise, which is how a thousand tiny lucky wins launder randomness into "progress." Every verdict writes a receipt sufficient to recompute it, and an independent auditor — a different model — reads each promotion and may dissent, on the record, without veto.

In three live runs, all published, the gate promoted Gemini's first proposal and refused its next two — every time. Most of those refusals were of measured improvements: real gains, turned down for falling under the margin. Those refusals are worth more than any promotion we could show you. A system that can only say yes to itself proves nothing by saying yes.

The process was the philosophy

The unexpected lesson: building this forced the same discipline onto us.

Our benchmark flattered us once — paired sampling made cutting measurement free, and the "efficiency gains" our evolver kept finding were partly an artefact. We found it by auditing our own results, measured both regimes, switched to the harder one, and published the comparison. Our figures drifted from the code once — a process-randomised hash quietly broke replayability — and the fix was to pin every published number to the code by test. Our chart design was reviewed by an independent reasoner that rejected our first axis for hiding real failure cases; it was right, and the correction is printed on the figure. Even the second axiom was refined under fire from its own author. And the collaboration that built all of it ran on the first axiom: an AI pair-engineer proposing at speed, a human judging, refusing, and redirecting — neither of us trusted as the judge of our own claims.

None of this was in the plan. All of it is the point. A method you only apply to your subject is a pose; applied to yourself, it becomes a practice.

Toward a thousand turns

Today the system runs a few gated turns of self-improvement, demonstrated end to end.

The architecture exists so that a thousand turns would still be falsifiable: margins against noise, auditors against metric-gaming, receipts for lineage, and — the quiet advantage of synthetic worlds — an inexhaustible supply of fresh, unseen tests, which lets repetition remain science instead of becoming p-hacking.

Free for everyone, human or AI. That is not a licence footnote; it is the thesis restated. Science is the practice of claims that anyone may check. We tried to build software the same way.

Ex probatione propria non sequitur veritas. — including ours.

— Marco Vanadia (mova), built meta-agentically with an AI pair-engineer, August 2026

Try it yourself