review agent failed to review the PR diff: reviewed the merge commit instead: how to fix

TL;DR

Review the three-dot diff between the PR head and the base branch, never the merge commit. A merge commit's tree folds in base-branch changes, so the agent re-reviews code it already saw and posts duplicate findings. Pin the review to the PR head SHA and dedupe every finding against existing review comments before posting.

The problem

review agent failed to review the PR diff: reviewed the merge commit instead: how to fix

When to use

• Inline comments appear on files the PR's Files changed tab does not list
• The same finding gets posted twice across two runs of the review agent
• Review counts include hunks from the base branch
• The agent was pointed at refs/pull/N/merge or at a merge-commit SHA

When not to use

• Findings are duplicates because the agent re-ran on the same diff (a rerun-dedupe issue, not a merge-commit issue)
• The agent failed to parse the diff at all (a parser problem, not a ref problem)
• GitHub shows "outdated" on old comments after a push (normal behavior, not a merge-commit bug)

Steps

1. Confirm which ref the agent actually reviewed

Check the agent's config or logs for the ref it fetched. Look for refs/pull/123/merge, a /merge suffix, or a merge-commit SHA.

gh pr view 123 --json headRefOid,baseRefName --jq '{head: .headRefOid, base: .baseRefName}'

Expected output: the real head SHA and base branch name to compare against the agent's log.

2. Reproduce the bad diff locally

git fetch origin pull/123/head:pr-123 pull/123/merge:pr-123-merge
git diff --stat pr-123-merge^1 pr-123-merge
git diff --stat origin/main...pr-123

Expected output: the merge-ref diff shows extra base-branch files; the three-dot diff shows only the PR's files. That gap is the duplicate source.

3. Repoint the agent at the head ref

Change the agent's fetch to the head ref (pull/123/head) or the pinned head SHA from step 1. Never hand it the /merge ref.

Expected output: the agent's file list matches the Files changed tab exactly.

4. Dedupe findings against existing review comments before posting

gh api repos/OWNER/REPO/pulls/123/comments --paginate -q '.[].path' | sort | uniq -c | sort -rn | head -20

Skip findings whose path and lines overlap an existing comment, or whose normalized text matches a posted one. Expected output: zero re-posts; only net-new comments go out.

5. Record the reviewed head SHA on the review

Put the reviewed head SHA in the summary footer or run metadata. Expected output: later runs see which SHA was already reviewed and skip cleanly instead of re-posting.

Tool compatibility

• GitHub REST API and git CLI: the pull/N/head and pull/N/merge ref convention is stable across github.com and GHES.
• gh CLI 2.x: pr view --json headRefOid,baseRefName works on all recent releases.
• Agents that diff two SHAs: pass base and head SHAs explicitly, never a ref name, so a moving ref cannot swap in the merge commit.

Variant phrasings

review bot commented on code outside the PR

Same root cause in most cases. Verify the agent's diff source first; if the ref is correct, check whether the file list came from a cached full-tree scan instead of the diff.

duplicate findings across review runs

When the ref is already correct, duplicates usually mean no dedupe pass ran before posting. Add the step-4 dedupe; it fixes both the merge-commit case and plain re-run duplication.

agent reviewed base branch files in the PR

Confirm with the step-2 reproduction. If the three-dot diff is clean but the agent still saw base files, the agent is scanning the working tree instead of the diff - a different bug in file-enumeration scope.

Why it happens

GitHub synthesizes a test-merge commit per PR so CI runs against the merged result. Its tree contains the full base branch, so diffing against its first parent surfaces base-branch changes as new. Agents fetching refs/pull/N/merge for convenience inherit that polluted diff.

Edge cases

• Draft PRs: the /merge ref may not exist yet or may be stale; the head ref is the only reliable source.
• Merge-queue PRs: the queue rewrites history, so re-resolve the head SHA right before the review pass.
• Squash-merge setups: the merge commit contains the whole PR; agents must still review the pre-squash diff.
• Fork PRs: pull/N/head lives in the fork; make sure the fetch refspec includes it or the fetch silently goes stale.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Vq0qYXV6WvsYyNtesoyGlw