- Book: RAG Pocket Guide
- The series: Pocket Guides for Developers — 8 short books on agents, RAG, prompts, observability, events, databases and system design — the whole series
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
Your coding agent opened the pull request an hour ago. Since then, two reviewers have left nine line comments on it: a missing null check, a test name that no longer matches what the test does, a query that skips the index. So you open the branch, copy the first comment into the agent's terminal, wait, scroll back to GitHub, find the second one, copy it, and repeat seven more times.
GitHub already stores every one of those comments with its file, its line range and the diff hunk it was written against. That is everything an agent needs to act on it. So pull them out once, turn them into one prompt file, hand the file to the agent on the PR's branch, and then check mechanically that every commented range actually changed.
The loop below uses gh, jq, one call to Claude Code or Codex, and git diff. At the end I show the in-app version in Hermes IDE. I build Hermes, so weigh that section accordingly.
Step 1: pin the PR's branch in its own worktree
Review comments point at line numbers in one specific commit: the head of the PR when you fetch them. If the agent works on a different commit, every line number is off. The first job is to check out exactly that commit, in a folder of its own, away from whatever you have open in your main checkout.
pr=42
kit=~/code/review-loop
out="$kit/pr-$pr"; mkdir -p "$out"
read -r branch head < <(gh pr view "$pr" \
--json headRefName,headRefOid \
--jq '[.headRefName, .headRefOid] | @tsv')
git fetch origin "$branch"
git worktree add "../pr-$pr" "$branch"
cd "../pr-$pr"
test "$(git rev-parse HEAD)" = "$head" || echo stale
headRefName and headRefOid are both fields of gh pr view --json. When the branch only exists on one remote, git worktree add creates a local tracking branch for it. If the script prints stale, your local branch was behind the PR, so run git pull --ff-only in the worktree before going on. And if the agent's own worktree still has the branch checked out, git refuses to check it out a second time: run everything from that folder instead.
The prompt, the raw JSON and the scripts live in ~/code/review-loop, outside the worktree, so the agent never edits them and they never end up in a commit. This also assumes the branch lives in your repository, which is the normal case for an agent's PR.
The review comments already know where they belong. The loop just moves them.
Step 2: pull every unresolved thread with gh api graphql
The REST endpoint that lists review comments gives you path, line, start_line and diff_hunk for each comment, but no field that says whether its thread was resolved. That state lives on the review thread, which the GraphQL API exposes as PullRequestReviewThread, with isResolved and isOutdated on it. Save this as ~/code/review-loop/threads.graphql:
query($owner: String!, $repo: String!,
$pr: Int!, $endCursor: String) {
repository(owner: $owner, name: $repo) {
pullRequest(number: $pr) {
reviewThreads(first: 50, after: $endCursor) {
pageInfo { hasNextPage endCursor }
nodes {
id isResolved isOutdated subjectType
path diffSide line startLine
comments(first: 20) {
nodes {
authorAssociation body diffHunk url
}
}
}
}
}
}
}
Then fetch every page and keep the open threads:
gh api graphql --paginate --slurp \
-F owner='{owner}' -F repo='{repo}' -F pr="$pr" \
-F query=@"$kit/threads.graphql" \
> "$out/raw.json"
jq '[.[].data.repository.pullRequest
| .reviewThreads.nodes[]
| select(.isResolved | not)]' \
"$out/raw.json" > "$out/open.json"
From the gh api manual: --paginate on a GraphQL query needs the $endCursor variable and pageInfo { hasNextPage endCursor }, which is why both are in the query. --slurp wraps all pages in one array. -F fills {owner} and {repo} from the repository you are in, sends pr as an integer rather than a string, and reads a value from a file when it starts with @.
A few fields matter later. line is the last line of the thread and startLine the first, set only for multi-line comments. subjectType is LINE or FILE, because a comment can target a whole file. diffSide is LEFT or RIGHT. authorAssociation says how the commenter relates to the repo: OWNER, MEMBER, COLLABORATOR, CONTRIBUTOR, NONE and a few more (enum list).
Step 3: turn the threads into one prompt with jq
Save this as ~/code/review-loop/prompt.jq:
def trusted:
.authorAssociation
| IN("OWNER", "MEMBER", "COLLABORATOR");
def where:
if .subjectType == "FILE" then "whole file"
elif .startLine then "lines \(.startLine)-\(.line)"
else "line \(.line)" end;
def quote: split("\n") | map("> " + .) | join("\n");
[ .[]
| .comments.nodes |= map(select(trusted))
| select(.comments.nodes | length > 0)
| select(.isOutdated | not) ]
| "Review comments on PR #\($pr). For each one,",
"change only what it asks. Do not commit, push",
"or edit files no comment mentions. End with one",
"line per comment: its number and what you did,",
"or why you changed nothing.",
"",
( to_entries[]
| .value as $t
| "## \(.key + 1). \($t.path), \($t | where)",
"",
($t.comments.nodes | map(.body | quote)
| join("\n>\n")),
"",
"~~~diff",
$t.comments.nodes[0].diffHunk,
"~~~",
"" )
And build the file:
jq -r --arg pr "$pr" -f "$kit/prompt.jq" \
"$out/open.json" > "$out/prompt.md"
Each comment becomes a numbered section with the path, the line or range, every reply in the thread quoted, and the diff hunk the reviewer was looking at. The numbers matter: the agent's closing summary refers to them, so you can match each of its lines to a section of the prompt.
Review comments are text from other people, and you are about to give that text to an agent that can edit files. On a public repository anyone with read access, which is everyone, can review and comment. The trusted filter keeps only owners, members and collaborators, so a stranger's comment stays out of the prompt. It is one filter. The agent still reads the code and anything a trusted account writes, so it does not stop prompt injection on its own. Loosen it on purpose if your team works differently. Don't just delete it. IN needs jq 1.6 or later.
Step 4: run the agent on the worktree
From inside ../pr-42, with Claude Code:
claude -p "Address the review comments in the input." \
--permission-mode acceptEdits \
< "$out/prompt.md" | tee "$out/agent.txt"
Or with Codex:
codex exec --sandbox workspace-write - \
< "$out/prompt.md" | tee "$out/agent.txt"
claude -p reads piped stdin alongside the prompt you pass, and acceptEdits lets Claude write files without asking. Shell commands outside the read-only set still need an allow rule, so add something like --allowedTools "Bash(npm test *)" if you want it to run your tests (headless docs). The same page recommends --bare for scripted calls, but bare mode skips your CLAUDE.md, and here you probably want the project's rules loaded. codex exec - reads the whole prompt from stdin, and since it runs in a read-only sandbox by default, --sandbox workspace-write is what lets it edit. Codex prints only its final message on stdout, which is the per-comment summary you asked for.
The prompt tells the agent not to commit, so its changes stay in the working tree. The next step can then compare them against $head, the commit the threads' line numbers refer to, and you decide what gets committed.
Changed lines are easy to list. Whether the change answers the comment is still your call.
Step 5: check that every commented range changed
git diff -U0 prints only the changed lines, and each hunk header says which lines of the old file it replaced. In the unified format the count is left out when a hunk covers one line, and an empty hunk sits at the line before it. Old-file line numbers are what the review threads use: for a thread that isn't outdated, line refers to the PR's current head, which is the $head you pinned in step 1. Save this as ~/code/review-loop/check.sh:
#!/usr/bin/env bash
# usage: check.sh <head-sha> <open.json>
head=$1
jq -r '.[]
| select((.isOutdated | not)
and .subjectType == "LINE"
and .diffSide == "RIGHT")
| [.path, (.startLine // .line), .line]
| @tsv' "$2" |
while IFS=$'\t' read -r path from to; do
hit=$(git diff -U0 "$head" -- "$path" | awk \
-v f="$from" -v t="$to" '
/^@@ / {
split($2, o, ",")
s = substr(o[1], 2) + 0
n = (2 in o) ? o[2] + 0 : 1
e = (n == 0) ? s : s + n - 1
if (s <= t && e >= f) hit = 1
}
END { print (hit ? "changed" : "UNCHANGED") }')
printf '%-9s %s:%s-%s\n' "$hit" "$path" "$from" "$to"
done
Run it from the worktree with bash "$kit/check.sh" "$head" "$out/open.json". git diff "$head" compares that commit with the working tree, so it sees the agent's uncommitted edits.
Treat the output as a reading list. changed only says the agent edited lines inside that range; whether the edit does what the reviewer asked is for you to read. Some UNCHANGED rows are fine: "add a test for this" gets answered in another file. Put agent.txt next to it and read the UNCHANGED ones first. The check also covers threads the trusted filter held back, so those usually show up as UNCHANGED.
When you are happy, commit and push:
git add -A
git commit -m "Address review comments on #$pr"
git push
If your team lets the author close threads, the resolveReviewThread mutation takes the thread id you already have in open.json:
gh api graphql -F id="$thread_id" -f query='
mutation($id: ID!) {
resolveReviewThread(input: {threadId: $id}) {
thread { isResolved }
}
}'
Do that one thread at a time, after you have read the change. A script that resolves everything the agent touched hides the comments it got wrong.
The edge cases that break the naive version
Outdated threads. A thread is outdated when a later push changed the lines it points at (isOutdated), and its line is nullable in the schema, so don't count on a number. The diff hunk then shows code that is no longer there. The jq filter keeps these away from the agent. List them for yourself:
jq -r '.[] | select(.isOutdated)
| "\(.path) \(.comments.nodes[0].url)"' \
"$out/open.json"
Multi-line comments. A reviewer who drags across lines 40 to 52 produces startLine: 40 and line: 52. A script that reads only line checks one line out of thirteen. Both the prompt and the check use the full range.
File-level comments. subjectType: FILE has no line at all. The prompt labels it "whole file", and the check skips it, because any edit to the file would count.
Left-side comments. diffSide: LEFT means the reviewer commented on a deleted line, numbered in the base version of the file. Those numbers don't match the head commit, so the check skips them. Read those by hand.
Long threads. The query fetches 20 comments per thread and 50 threads per page. --paginate pages through the threads but not through the nested comments, so a thread with a longer argument than that gets cut. Raise the number if your reviews run long.
The in-app version: Hermes's Review Desk
The gh loop handles comments that arrive on GitHub, after the PR exists. Most of that review can happen earlier, on your machine, before you open the PR at all. That is the part Hermes does.
Hermes saves every agent turn, so you can diff or restore any single turn, with undo. The Review Desk (⌘G, Ctrl+Shift+G on Windows and Linux) shows the changes by turn or by file, with risk flags, and a line comment you leave there goes back to the agent that wrote that line (README). No copying a comment into the wrong terminal, no prompt file. When the work looks right, Land commits, opens a pull request or merges locally, into the branch the task started from.
The Review Desk (⌘G) in By turn view for a Claude Code task in orders-api: two turns, turn 1's diff of src/db.ts, the changed files and a commit message drafted from the turns.
The two fit together. Review locally in Hermes before the PR, then run the gh loop for whatever your teammates add on GitHub afterwards.
If this was useful
Most of this post is a small retrieval problem: out of a whole pull request, pick the exact context an agent needs (the file, the range, the hunk, the thread) and nothing else. That is the same discipline the RAG Pocket Guide covers for documents: chunking, filtering and ranking so the model sees the right few pieces. The series has eight short books in total, on agents, RAG, prompts, observability, events, databases and system design, and you can see all of them here.
Hermes IDE is free to download for macOS, Windows and Linux, source-available under BSL 1.1 on GitHub, and not affiliated with Nous Research, Anthropic, OpenAI or any other agent vendor.
