One morning my night report said the same thing about two projects, six times over:
- work is committed but the push failed, manual push needed
I went looking for the work. There wasn't any. The children had died in the first second of their lives, a harness binary had moved the day before and was no longer on the path, and the runner had reported a push failure for sessions that never started. The next night the children ran properly, committed real work in four projects, and the runner deleted all of it. Same message.
The night job had just gained a merge-request flow. Each child works in a throwaway worktree on a per-night branch; when it commits, the runner pushes the branch and opens a merge request for my morning. Two defects were stacked under that one sentence, and the second is the one worth the essay.
child, in a throwaway worktree runner
commits on the night branch
│
▼
did it commit? compare with upstream
✗ fresh branch, no upstream: every
child reads as "has unpushed work"
push, from the worktree
✗ push URL disabled there, by design
report: push failed, manual push needed
cleanup: remove worktree, delete branch
✗ the only ref to the commits is gone
The first was the test for "did the child commit anything". It compared the branch against its upstream. A fresh branch has no upstream, the lookup failed, the fallback value read as "has unpushed work", and from then on every child, working or dead, looked identical to the runner. A proxy for the fact, not the fact.
The second was the push itself. My worktrees have their push URL disabled by design, it is the rail that guarantees an unattended agent can never push, so the push ran from exactly the place that cannot push. It failed every night it was tried. The report said manual push needed. And then the cleanup step, which had never been told about any of this, removed the worktree and deleted the branch, the only reference to the night's commits. The runner destroyed the thing it had just asked me to save, and the morning report, which files a push failure under things I fix by hand after coffee, gave no sign there was nothing left to push.
What saved the work is git's habit of keeping unreferenced objects around until a garbage collection. The rescue is three commands, and the comment is the important one:
$ git fsck --no-reflogs --lost-found | grep commit
dangling commit 3f9c2a1d7e...
dangling commit 8b07d4c5a2...
$ git log -1 --oneline 3f9c2a1
3f9c2a1 drain: align the retry helper with the new timeout API
$ git branch rescue/2026-08-21 3f9c2a1 # a ref, nothing else
# do NOT gc, prune or worktree-prune first: unreachable is what gc
# deletes
Four repos came back that way the same night. A fifth had two orphans the sweep missed, found by hand while writing this; one was five days old, and a later merge message had described it as already on main.
The fix is three changes, and each is a version of the same rule: decide from what verifiably exists, never from what a message says.
-# did the child commit? a proxy: compare the branch with its upstream
-n=$(git rev-list --count "$b@{u}..$b" 2>/dev/null || echo 1)
+# did the child commit? the fact: HEAD moved while the child ran
+pre=$(git rev-parse HEAD)
+run_child
+post=$(git rev-parse HEAD)
+[ "$post" != "$pre" ] && committed=1
-# cleanup: always
-git worktree remove "$wt" && git branch -D "$b"
+# cleanup: only when the remote verifiably holds every commit
+n=$(git rev-list --count "$b@{u}..$b" 2>/dev/null || echo 1)
+[ "$n" -eq 0 ] && git worktree remove "$wt" && git branch -D "$b"
Outcome is measured as HEAD movement during the child's run. A child that dies within thirty seconds with no commits is reported as exactly that, with the last two lines of its own output quoted in the report, and it gets no more rounds that night; a real session takes minutes even when it finds nothing to do. The push goes from the parent checkout, where pushing is allowed. And the branch is deleted only when the count of commits the remote lacks is zero, or the commits never existed. "Push failed" now means a commit exists and the remote does not have it, which is the only thing it ever should have meant.
The lesson generalises past git. Every night produces a report, and I had been treating the report as the state. It is a claim about the state, made by the same program that made the mistake.
- The destructive step in any automation has to be conditioned on the verified success, not the reported one.
And the two failures that look alike from outside, never started and started-then-failed, need different words, because the operator does different things for each. I had written that rule down for agents: re-verify before you repeat, assert from the primary source. It applies to shell scripts just as well, and they are even less likely to feel the tell.
