Scope note (read first): this describes a system I built and operate to develop an unannounced game. The game's identity, mechanics, and assets are withheld, and so are the harness's tuned prompts, gate implementations, and internal failure specifics. What's shown here is the method and the evidence discipline — the transferable part.
Summary
Over about four months (spring–summer 2026) I built and operated, solo, an operator-supervised multi-agent development harness that builds a real, shipping product. Its defining property isn't speed — it's that nothing an agent produces closes without machine-checkable proof, and no irreversible action happens without a human. This is a case study of the system and the evidence trail it leaves.
The idea
LLM coding agents are fast and unreliable. The engineering problem isn't getting output — it's trusting it. So the harness is built around one rule: an agent's work is unverified until a gate proves it. Coordination is automated; consequences are gated. And it's built to ship, not to gold-plate: every gate exists so I can move fast without shipping something broken — verification in service of velocity, not instead of it.
Architecture (concept level)
- A five-role loop: Strategy → Execution → Critic → Eval → Ops. Judgment roles (Strategy, Critic, Eval) run on stronger models; execution roles on cheaper ones — cost follows the difficulty of the decision, not a flat default.
- A manager / orchestration layer. Above the execution agents sits one orchestration role that I direct — it plans each unit of work, routes it to the right role and model, and holds the system's state between steps. I designed the roles, the gates, and the routing; the harness runs them. I'm not outside the loop supervising a black box — I'm the system's judgment and authority, and the manager is the layer that extends that across many parallel agents.
- A cold, independent critic gate. Before a consequential change closes, it's reviewed by a Critic running on a fresh, zero-context session — a different strong model with no memory of how the code was written — so it reviews the work itself, not the author's rationale for it. It can send the change back for rework. A self-review rubber-stamps; a cold critic catches what the author already talked themselves past.
- A human-in-the-loop autonomy ladder: the loop's handoffs are automated — one role hands to the next without me — but every irreversible act (deploying a build to a device, committing to git) stays behind an explicit human approval. Automate coordination; never automate the irreversible.
- A differential oracle for correctness: the core logic is implemented twice and the two versions are fuzzed against each other. Where they disagree, one is wrong — no gold labels required.
The evidence discipline (the differentiator)
Every closed unit of work leaves a durable, machine-checkable proof:
- NO-PROOF-NO-CLOSE gate. A work item cannot close until an automated check confirms its proof exists on disk. The loop physically cannot skip it.
- Provenance-bound proof. On-device validation screenshots are sanitized (sensitive regions blacked out), and provenance manifests bind images to the exact git SHA, screen dimensions, and redaction method that produced them — so an artifact traces back to the commit it proves.
- Human-gated checkpoints. Each checkpoint records scoped git staging (explicit paths only), a commit/SHA trail across the repos it touches, an artifact-registry audit, and an explicit operator approval.
- Periodic self-evaluation. An independent evaluation role produces a numeric health score with a delta versus the prior period and a failure taxonomy; regressions feed a failure registry that drives fixes.
The testing oracle
The product's core logic is held to property-based invariant tests — generated inputs are thrown at the engine and a set of invariants must hold for every one (e.g. a detector must agree with an independent full re-scan, and detection must be side-effect-free). The suite runs against the authoritative implementation, so an invariant is enforced on the logic, not asserted in prose (last run: zero failures). As a standalone, fully public demonstration of the same technique, my match3-engine repo carries 16 jqwik property invariants over random inputs.
Operating record (Apr–Jul 2026, from the on-disk archive)
~200 completed work-arcs · ~190 human-gated checkpoints · 74 independent critic reviews · 13 periodic self-evaluations · a growing failure registry with per-item root-cause fixes · a ~200-file sanitized proof archive with ~90 provenance manifests.
Verifiable outcomes (all public)
- An arcade game — Tap Dodge Rush, under SeraphLight Studios — shipped end-to-end to Google Play.
- A one-character bug fix merged upstream into TeaVM (the Java-to-JavaScript compiler), closing a long-dormant issue.
- A live public model-drift board grading 16 LLMs daily on a frozen, deterministically-graded suite — no LLM-as-judge, so a score change is real.
- Ten public repos, including a differential-oracle testing project and a Model Context Protocol server built from the spec.
What I'd bring to a team
Treat AI output as unverified until proven. Build the gate before the feature. Make failures loud, not silent. Keep a human on the irreversible path. The discipline transfers to any codebase — the harness just made me practice it a few hundred times.
Full architecture case study & repo: github.com/egnaro9/agentic-dev-harness · Portfolio: egnaro9.github.io
