Last time I wrote about building triage-lens, a vulnerability triage CLI, by delegating all implementation to Claude Code despite not being able to read code. This is the sequel. For Phase 2, I had the AI do everything unattended, overnight, while I was asleep.
When I woke up, the test suite had grown from 145 to 330, an adversarial review by a second AI was already done along with the fixes, and one pull request sat waiting, unmerged. Also, at some point during the night, a PR had deadlocked and been quietly rescued without me.
Here is the record of that night.
(Note: this is a translation of my Japanese article on Zenn. The tool's reports support Japanese and English as of v0.2.0.)
Before bed: replacing approval with a contract
Until now, our process had one central rule: present a plan, get my approval, then implement. That gate is where quality comes from. But I can't approve anything while asleep, and dropping the gate entirely felt reckless.
What I believed going in was that risk control lives in the instructions and the design, so I checked the prompt carefully before handing it over. In place of the approval gate, I wrote four things into it.
- A frozen scope. Tonight's allowed work: CycloneDX input support, English reports, README updates. Anything else was explicitly forbidden.
- Hard lines. No direct pushes to main. No merging. No syncing to the public repository.
- A rule for doubt. If a requirement was ambiguous, err on the side of not implementing, and leave the question in an issue. If blocked by an error, push the work-in-progress and describe the situation.
- A morning report format. What it can now do (three lines, for a non-engineer), verification commands I can copy and paste, test results, and whether anything needs my judgment.
In short: instead of asking me mid-flight, leave a trail of your decisions. The approval didn't disappear. It became something I could grant after the fact, over coffee.
The morning report: the fixes scared me more than the numbers
The report was waiting when I woke up. 330 tests green, CI green on three Python versions, PR unmerged. Exactly as instructed.
The interesting part came next. After finishing the implementation, Claude Code had followed the instructions and handed the entire codebase to a different AI (the codex CLI) for an adversarial review. Seven findings came back. Rather than accepting them at face value, it reproduced each one, fixed the four that were in scope, and filed the three debatable ones as issues for me to rule on later.
Two of the fixed defects were the kind a security tool cannot afford.
One: the tool had been reading "unaffected version" records as "fixed versions," which in some conditions produced output that recommended downgrading to an older release. Two: the input format check was loose enough that Trivy output could be misclassified, yielding a clean report with zero findings. Both errors run in the same direction, making dangerous things look safe, and both belong to the same family as the defect from my previous article, where a missing exploit probability was displayed as "low."
Three builds, three defects of this exact family. At this point I stopped treating it as coincidence and started treating it as the default suspicion to bring to any AI-written implementation.
The deadlock: handling what I couldn't understand
Buried in the report was this passage. Deleting a branch had auto-closed an unmerged PR, which led to a trap: a closed PR can't change its base branch, and a PR whose base branch is gone can't be reopened. So the AI temporarily restored the branch, reopened the PR, repointed its base, and deleted the branch again.
Honest admission: reading this first thing in the morning, I did not understand what had happened. People fluent in Git operations probably find this routine. I had no way to judge it.
What let me keep reading was the shape of the report: what happened, why, how it was recovered, and what the current state is, ending with claims I could verify (same PR number, still open, based on main, mergeable). I couldn't evaluate the content, but I could evaluate whether the form was complete. When someone who can't read code manages an AI's work, that form is what you lean on. Later I had the AI break the incident down until I could explain it in my own words, and only then felt settled about it.
Round two: making the tests prove they catch the bug
In my morning ruling, I picked one filed issue to fix before anything went public. When the same vulnerability was detected in two places, deduplication kept whichever came first, so under the wrong conditions the higher CVSS score could silently vanish. That understates risk, which put it in the must-fix family.
What impressed me about the fix PR was how the tests were validated. Passing tests after a fix proves little, so I had it deliberately revert the implementation to the old first-wins behavior and confirm that 17 tests fail. Do the tests actually catch the defect, or were they merely written to pass? You don't need to read code to demand that check.
One more decision I've grown fond of. We added a rule that when two records disagree on the fixed version, the newer one wins. But version comparison was limited to plain digits-and-dots notation like 1.2.10. Trying to be clever about Debian's epoch-prefixed strings and similar formats risks inventing a false ordering, so anything uncomparable falls back, safely, to "unknown." Declining to handle everything intelligently is also a security decision.
The suite ended at 373 tests, and v0.2.0 is now public.
What can run unattended, and where a human must stand
My conclusion after one night: where you draw the line depends on what you refuse to compromise.
For me, that was "never make danger look safe." It's the one lie a triage tool must not tell, so that property gets verified with a human involved. Everything protecting it, the second-AI review, the reproduction tests, the report format, went into the contract, and once it was there, the implementation itself was free to proceed while I slept.
If this were a writing project instead, the thing a human would guard is whether the piece stays worth reading. Deciding what to hand to the machine and where to keep your own eyes: that design work is the actual job of the person doing the delegating.
The repository is public. Next time I'll write about the tool's substance: what goes wrong when you rank vulnerabilities by CVSS alone, and how EPSS and CISA KEV change the picture.
