A merged PR. A one-line summary. The changelog says "improved error handling."
Can anyone verify that sentence next quarter? Usually not.
I keep meeting teams that draft release notes with a model. The pipeline is fast. The audit trail is gone.
So here is a myth FAQ about agent-written changelogs. Four claims, the evidence for each, and a corrected mental model.
Myth 1: "The diff already tells you why"
The diff tells you what changed. It rarely tells you why.
Why lives in the commit body, the linked issue, and review comments. Those are separate stores with separate retention rules.
Check what your repository actually keeps:
git log --no-merges --format='%H %s%n%b' v1.4.0..HEAD | head -40
If commit bodies are empty, your model is guessing. It generalizes from the diff. That is invention with good grammar.
Corrected model: a changelog claim needs a citation, not a vibe. Anchor every claim to a commit SHA.
Myth 2: "The model read the whole PR"
It read what your tool passed it. That is often a truncated diff.
Large diffs get chunked, clipped, or dropped. Generated lock files eat the context budget first.
A one-minute test
- Pick a PR with 2,000+ changed lines.
- Ask for a summary with line numbers.
- Check five cited line numbers against the real diff.
- Count how many are wrong or missing.
If citations are absent, you cannot tell omission from correctness. Silence looks a lot like accuracy.
Corrected model: require line or commit anchors. Treat unanchored prose as unverified, not as true.
Myth 3: "Green CI means the changelog is accurate"
CI tests code. It does not test prose.
A pipeline can pass 4,000 tests while the release note claims a behavior change that never shipped. No test asserts on your changelog sentence.
Corrected model: changelog accuracy is a separate check. Make it a test with a real failure mode.
Myth 4: "One prompt is a pipeline"
One prompt is one function call. Pipelines need idempotency, retries, and a schema.
Ask yourself: what happens on a re-run? Do you get the same entries, or a second set? Does a failed batch leave half-written notes?
Corrected model: generate claims as data first. Render prose last.
The artifact: an evidence file plus a claim gate
Here is the shape I would use. Three steps, one gate.
Step 1: extract evidence, not prose
# scripts/changelog-evidence.sh (proposal; run it on your own repo)
set -euo pipefail
RANGE="${1:?usage: changelog-evidence.sh <from>..<to>}"
echo "## commits"
git log --no-merges --reverse \
--format='sha=%H%nauthor=%an%ndate=%aI%nsubject=%s%nbody=%b' "$RANGE"
echo "## files"
git diff --numstat "$RANGE"
This output is deterministic. Same range, same bytes. No model is involved yet.
Step 2: classify into a fixed schema
Ask your endpoint for JSON, one object per commit:
{
"sha": "a1b2c3d",
"section": "fixed",
"claim": "POST /v1/jobs returns 409 on duplicate keys",
"confidence": "high"
}
Keep the sections small: added, changed, fixed, removed, internal.
Step 3: gate the claims
Every claim must carry a commit anchor. An unknown SHA means reject.
# scripts/check_claims.py (proposal; not executed here)
import re, subprocess, sys
ANCHOR = re.compile(r"\(([0-9a-f]{7,40})\)$")
def shas(range_spec):
out = subprocess.check_output(
["git", "log", "--format=%H", range_spec], text=True)
return [line.strip() for line in out.splitlines() if line.strip()]
def main(path, range_spec):
known = shas(range_spec)
bad = 0
for n, raw in enumerate(open(path, encoding="utf-8"), 1):
line = raw.strip()
if not line.startswith("- "):
continue
m = ANCHOR.search(line)
if not m or not any(s.startswith(m.group(1)) for s in known):
print(f"line {n}: unanchored or unknown commit")
bad += 1
print(f"{bad} unverified claim(s)")
return 1 if bad else 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1], sys.argv[2]))
Run it in CI as a required check. A failing gate blocks the draft, not the release.
The claim file then looks like this:
- [verified] `POST /v1/jobs` returns 409 on duplicates (a1b2c3d)
- [unverified] improved error handling (no anchor)
- [rejected] retry cap changed; no commit in range touches retry code
Where a free tier fits, and where it does not
The classification step is small and repetitive. It fits a free endpoint well.
If you want to try that step without buying a key, MonkeyCode's free model access and free server option are one place to evaluate. I am not asserting quotas, model names, or hardware here. Check those yourself before you design around them.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Keep the gate on your side. Never let a free endpoint own the pass/fail decision.
Limitations
- The gate checks provenance, not truth. A cited commit can still describe wrong behavior.
- Squash-only, one-line commit histories give the classifier almost nothing to work with.
- Reverts and silent removals are easy to miss on the first pass.
- It adds a review step. That is the whole point, but it costs reviewer time.
- If your notes go through legal or marketing, automate the draft only, never the sign-off.
Who should not use this
Skip it if your changelog is legal copy. Skip it if compliance signs exact wording.
Skip it if nobody on the team will read a diff. A gate without a reader is theater.
The corrected mental model
A changelog is a set of verified claims with citations. A model can draft and classify. It cannot certify.
So before your next release: which sentence in your notes has no commit behind it?
