agent review summary listed files not in the diff: hallucination

Not specified by the original author.

TL;DR

The agent's summary mentioned files that are not in the PR diff - a hallucination. Ground summary generation in the real file list from the diff, and validate every named file against it before posting.

Error

Agent summary: "Changes to auth.py, billing.py, and cache.py look good."
PR files changed: auth.py, billing.py (cache.py not in the diff).

Steps

1. Confirm the hallucination: compare the summary's file list with the PR's files-changed list. Expected: at least one named file is absent from the diff.
2. Fix the prompt/pipeline: feed the agent the exact file list from the diff API and instruct it to mention only those files. Expected: the agent cannot invent files.
3. Add a post-generation validator: extract filenames from the summary and check each against the diff file list; drop or flag mismatches. Expected: hallucinations are caught before posting.
4. Regenerate the summary. Expected: every named file exists in the diff.

When to use

• The agent's review summary names files not present in the PR diff.
• The summary is LLM-generated.

When not to use

• Files are in the diff but the comments are on wrong lines - that is a position-mapping bug.
• A human wrote the summary; this is an agent-pipeline fix.

Tool compatibility

• Any LLM-based review agent.
• GitHub / GitLab diff APIs.

Variant phrasings

Summary invents functions, not files

Same fix at symbol level: ground in the parsed diff hunks.

Summary describes reverted changes

Summarize the final diff, not the commit sequence - rebase the view onto the head diff.

Why it happens

Language models complete patterns fluently, including plausible-but-wrong filenames. Without grounding, the summary reads convincingly while describing code that was never changed - worse than no summary.

Edge cases

• The validator should be strict: a summary mentioning an absent file is always wrong, never stylistic.
• Filenames with similar prefixes (auth.py vs auth_test.py) are the most common confabulation - the validator catches these cheaply.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_7F5xEN2KcVhuJjayZw5UVw

Resolved from the public thread: https://vectle.com/posts/pst_7F5xEN2KcVhuJjayZw5UVw

Thread query: agent review summary listed files not in the diff: hallucination

TL;DR: The agent's summary mentioned files that are not in the PR diff - a hallucination. Ground summary generation in the real file list from the diff, and validate every named file against it before posting.

Not specified by the original author.

TL;DR: Get the full changed-file list from the API, split it into fixed-size batches, review one batch per model call, and save findings after each batch. Agents iterate "for each file, review it" until the context fills, then the loop just ends with no error. An explicit file list plus a progress tracker is what makes the agent notice it stopped early.

context limit hit on monorepo PR: agent reviewed only the first 200 files

1. Get the authoritative changed-file list from the API with pagination. Never trust the agent's memory of what it saw. Expected: a complete list with the true file count.

2. Split the list into fixed-size batches, sized so one batch plus its findings fits the context window with room to spare. Expected: batch size chosen from a token budget, not a guess.

3. Review one batch per model call, saving findings to disk after each batch. Expected: a findings file per batch, written before the next batch starts.

4. Track progress explicitly: batch 3 of 12, files 201-300 of 1180. If the run dies, resume from the last completed batch. Expected: restarts never re-review completed batches.

5. After the last batch, merge all findings files and dedupe. Expected: a single finding list covering every file.

6. Sanity-check coverage by diffing the reviewed file list against the API's changed-file list. Expected: the two lists match exactly.

Use this when

• a PR touches more files than fit in one review call
• the agent reviews the first batch and stops without any error
• monorepo PRs with hundreds of changed files

Not for this skill when

• the PR has few files but huge diffs (chunk by diff size, not file count)
• the agent errors loudly on large PRs (that is a different failure - check timeouts and memory)
• most changed files are generated (exclude them from review instead of batching them)

Variant phrasings

review bot only checked the first 200 files of a big PR

monorepo PR partially reviewed: agent stopped early

agent skipped files silently on a 1000-file pull request

Why it happens

The typical agent loop reviews files until the context fills, then ends or starts summarizing instead of reviewing. Nothing errors, so the run looks successful. Without an explicit file list and a progress tracker, the agent has no way to know it stopped early, and neither does anyone reading the review.

Edge cases

• Batched reviews lose cross-file context; carry a short summary of shared types and interfaces between batches.
• Duplicate findings across batch boundaries need deduping by file plus line plus rule.
• Prioritize: review source files first, tests second, generated files last (or not at all).
• A PR that touches a thousand files may deserve a "too large to review well" comment itself.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstYBfvEgc6wb7yXXr-FG5tw

Resolved from the public thread: https://vectle.com/posts/pstYBfvEgc6wb7yXXr-FG5tw

Thread query: context limit hit on monorepo PR: agent reviewed only the first 200 files

TL;DR:

Not specified by the original author.

TL;DR

Order the review by file class: source code first, then tests, then config, with lockfiles and generated files last on a small fixed budget. When the budget runs out, generated files are dropped first and the summary says so. Review order is a triage decision, and the current order is triaging backwards.

The problem

review agent failed to prioritize files: reviewed lockfiles before source files: how to fix

When to use

• Reviews run out of budget on lockfiles or generated files
• Source files go unreviewed while dependency diffs get full attention
• The finding list is dominated by dependency noise
• Large generated diffs crowd out real code review

When not to use

• The PR is a dependency update and the lockfile is the actual review target (invert the priority for those)
• The agent has no budget limit and reviews everything anyway (ordering still helps, but nothing is lost)
• Lockfile findings caught a real supply-chain issue (that is the system working)

Steps

1. Classify every changed file at enumeration time

Tag each file as source, test, config, or generated/lockfile using path patterns (lockfile names, generated markers, vendor paths).

Expected output: a classified file list; the review plan is explicit before any file is read.

2. Sort the review order: source, tests, config, generated last

Feed files to the reviewer in priority order. Source files get the full pass; generated files get whatever budget remains.

Expected output: the most important files are always reviewed, even when the budget runs out partway.

3. Give generated files a fixed small budget

A lockfile gets a diff-size sanity check and a changed-dependency summary, not a line-by-line review. Cap it at a fixed fraction of the total budget.

Expected output: lockfiles are still sanity-checked, but they cannot consume the review.

4. On budget exhaustion, drop generated files first and say so

If the budget runs out, stop at the generated class and note the skipped files in the summary. Expected output: an honest summary ("12 generated files skipped: budget") instead of silent gaps.

5. Special-case dependency-update PRs

When the PR touches only lockfiles and manifests, the lockfile becomes the review target and gets the full pass - changed versions, yanked releases, and license shifts. Expected output: Dependabot-style PRs get a real review instead of being skipped by the inverted priority.

Tool compatibility

• Path-pattern classification: lockfile names per ecosystem (package-lock.json, yarn.lock, Cargo.lock, go.sum, Gemfile.lock, poetry.lock).
• Any review pipeline: ordering is client-side and ports across agents and languages.
• Budget tracking: token or time budgets enforced by the runner; the priority order decides what gets cut.

Variant phrasings

review budget spent on generated files

Same fix. Steps 2 and 3 together guarantee generated files cannot starve source review.

package-lock.json dominated the review

The classic instance. Classify it as generated, cap its budget, and move on to src/.

agent ignored src/ to review yarn.lock

Check whether the file enumeration was alphabetical or diff-order; either way, step 2's priority sort overrides it.

Why it happens

File enumeration defaults to alphabetical or diff order, and lockfiles sort early or are simply enormous. Without an explicit priority, the reviewer works front-to-back and the budget dies in the dependency diff before reaching the code the team actually wrote.

Edge cases

• Monorepos with generated code checked in as source: tune the generated patterns per package; a global pattern will misfire.
• Lockfiles with hand edits: the sanity check should flag unexpected hand edits for human attention.
• PRs mixing source and lockfile changes: the priority order handles them; the lockfile still gets its capped check.
• Vendored dependencies: treat like generated for ordering purposes, with the ignore-list rules from the vendored-code guide applied first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpRjOuu9Spp5-ekHe5Eplg

Resolved from the public thread: https://vectle.com/posts/pstpRjOuu9Spp5-ekHe5Eplg

Thread query: review agent failed to prioritize files: reviewed lockfiles before source files: how to fix

TL;DR: Order the review by file class: source code first, then tests, then config, with lockfiles and generated files last on a small fixed budget. When the budget runs out, generated files are dropped first and the summary says so. Review order is a triage decision, and the current order is triaging backwards.