Originally published at devopsdiary.blog. Post R24 in "The Quiet Years" series.
The draft title for this post has been sitting in my publishing calendar since April: "I Built 24 Repos in 60 Days." Every few weeks I'd glance at it and the number was wrong again. 24 became 30, then 41. The 60 days stretched into five months. I kept not writing it, which turned out to be the best editorial decision I never made, because somewhere in there the story stopped being the count.
The count is real, for the record. AIEOS (the framework I introduced a couple of weeks ago) is now 41 repositories: a 15-layer model of the software delivery lifecycle, a multi-agent harness, a browser console, a CLI guide named Sherpa, 13 tool adapters and a control plane that can walk an initiative unattended. Every repo runs under CI. The adapters produce signed Sigstore attestations. The orchestration core sits at 100% line coverage, with 452 tests passing on Windows and Linux both.
But the number worth asking me about is a different one: zero. That's how many times adapter conformance had passed when I audited it on July 1, behind two months of green checkmarks. The gap between those two numbers taught me more than the 41 did.
March in Omaha is good building weather
It's gray, it's 34 degrees and there's nothing else to do.
January was nine commits of poking at an idea. February was scaffolding. March was 358 commits across 24 repositories, 22 of which didn't exist on the 1st. By April 9, all eight pipeline layers were built. 580-some GitHub contributions for the year, nearly all of them inside that window. I've been shipping software for 30 years and I have never produced anything close to that rate. Nobody has, by hand.
The honest mechanics: I barely wrote code. I wrote rules, templates, prompts and validators, then let the model generate against them, with every artifact frozen before the next one built on top of it. So when people ask whether the pace everyone brags about with AI-assisted development is real, I can answer from the far side of it. It's real. That was never the question for me.
The question was the one I'd spent three years asking other teams: how do you know any of it works?
Two months of green, zero real passes
On July 1, I went to verify a single adapter that was supposedly finished, and something in its pipeline caught my eye. The signing step was skipped. Not red. Skipped.
Pull that thread across all 13 adapters and the picture comes apart. Every conformance job ran under continue-on-error: true, a setting that forces the step to report success no matter what actually happened. The signing steps were gated on a genuine pass, and they were skipped in every adapter's latest run. A gate like that only skips when its condition is false. Conformance hadn't passed. Not once. On any adapter. Ever. The dogfooding loop I'd been describing in my own docs did not exist, and nothing was ever going to tell me.
The root causes were almost insultingly ordinary. One adapter's CI job never installed pytest, so the adapter had nothing to run and failed all four criteria. The other eleven read input keys the test harness never sent them. KeyError, before doing any work, on every run, since late April.
Twelve days after that audit I published a post adopting Sonar's term "verification debt" for the gap between what an agent produces and what you've actually verified. I want it on the record that I was carrying two months of it while I wrote the definition.
The fix started with deleting the mask. The single highest-value change in the whole remediation was removing one line of YAML: a genuine pass now signs, and a genuine failure now breaks the build, loudly. Then came a harness redesign so each test suite declares its own input contract, plus real fixtures for adapters that need a container image or a live endpoint to prove themselves against. By July 5, all 13 adapters were producing signed conformance attestations from genuine passes.
BEFORE JULY 1 (masked)
conformance fails
|
v
continue-on-error: true
|
+--> step reports "success" --> green checkmark
|
'--> signing step (gated on a genuine pass):
SKIPPED, every run. No attestation, ever.
AFTER (unmasked)
conformance fails --> build breaks, loudly
conformance passes --> signed attestation uploaded
One YAML key was the difference between evidence and decoration. The skipped signing step was the only honest signal in the pipeline.
The green means something now.
The framework kept catching its author
v1.3 shipped in mid-July with a claim I liked: three drivers (Sherpa for guided sessions, the console for point-and-click, a "dark factory" for unattended runs) over one shared engine. Then I ran the first dogfood pass with a real model in the loop, and the run that passed also falsified the claim. The console required a flow file that exists nowhere in the framework except the console's own test fixture. It had never loaded a real kit. Three drivers over one engine was true for two of them, and my own validation run is the only reason I know.
Same run, worse find. The function enforcing freeze-before-promote, the framework's second commandment, had a full test suite and zero production callers. Tested thoroughly. Called never. The invariant held anyway, but only because an unrelated bug happened to block the same path. If "the safety property was upheld by a different defect" doesn't unsettle you, read it again slowly.
I added a Windows runner to CI the same week. It failed on an encoding gap 45 seconds after it started existing, which led to the suite's first fully correct Windows run ever: 435 tests, no environment crutches. Ten gaps closed inside two days after that, each proven closed with a failing test first, not asserted closed in a commit message. The register tracking them holds names, dates and evidence instead of adjectives.
What the pace actually bought
In May I wrote that the agent is 20% of the work and the platform is the other 80. I wrote it about other people's agents. It's true of mine too, and I built the platform side specifically so that 80 wouldn't be improvised.
When people ask what five months of full-speed AI-assisted building is like, this is the answer I give now. Generation is the cheap part, cheap enough that the repo count stops being interesting. A green checkmark is a claim somebody configured. Evidence is what survives an attempt to falsify it, and the distance between those two was one unreviewed YAML key. And a governance system you won't run on itself is a slide deck. Mine caught its own author four times in one July (masked conformance, a phantom driver, an unenforced invariant, an encoding bug), and each catch was embarrassing for about an hour and then permanently load-bearing.
The stale title treated the count as the accomplishment. Five months on, the count is the least interesting artifact in the org. What I'd actually show you is the July 1 audit that called my own checkmarks liars, and the pipeline changes that made lying structurally hard afterward. AI will happily build you 41 repositories. Whether you can trust them is still your job.
Todd Linnertz is a Senior Solutions Engineer with thirty years of enterprise engineering experience. He's the creator of AIEOS, an open-source AI governance system for software delivery teams. Find him at devopsdiary.blog and github.com/wtlinnertz.
