review agent failed to prioritize files: reviewed lockfiles before sou

Not specified by the original author.

TL;DR

Order the review by file class: source code first, then tests, then config, with lockfiles and generated files last on a small fixed budget. When the budget runs out, generated files are dropped first and the summary says so. Review order is a triage decision, and the current order is triaging backwards.

The problem

review agent failed to prioritize files: reviewed lockfiles before source files: how to fix

When to use

• Reviews run out of budget on lockfiles or generated files
• Source files go unreviewed while dependency diffs get full attention
• The finding list is dominated by dependency noise
• Large generated diffs crowd out real code review

When not to use

• The PR is a dependency update and the lockfile is the actual review target (invert the priority for those)
• The agent has no budget limit and reviews everything anyway (ordering still helps, but nothing is lost)
• Lockfile findings caught a real supply-chain issue (that is the system working)

Steps

1. Classify every changed file at enumeration time

Tag each file as source, test, config, or generated/lockfile using path patterns (lockfile names, generated markers, vendor paths).

Expected output: a classified file list; the review plan is explicit before any file is read.

2. Sort the review order: source, tests, config, generated last

Feed files to the reviewer in priority order. Source files get the full pass; generated files get whatever budget remains.

Expected output: the most important files are always reviewed, even when the budget runs out partway.

3. Give generated files a fixed small budget

A lockfile gets a diff-size sanity check and a changed-dependency summary, not a line-by-line review. Cap it at a fixed fraction of the total budget.

Expected output: lockfiles are still sanity-checked, but they cannot consume the review.

4. On budget exhaustion, drop generated files first and say so

If the budget runs out, stop at the generated class and note the skipped files in the summary. Expected output: an honest summary ("12 generated files skipped: budget") instead of silent gaps.

5. Special-case dependency-update PRs

When the PR touches only lockfiles and manifests, the lockfile becomes the review target and gets the full pass - changed versions, yanked releases, and license shifts. Expected output: Dependabot-style PRs get a real review instead of being skipped by the inverted priority.

Tool compatibility

• Path-pattern classification: lockfile names per ecosystem (package-lock.json, yarn.lock, Cargo.lock, go.sum, Gemfile.lock, poetry.lock).
• Any review pipeline: ordering is client-side and ports across agents and languages.
• Budget tracking: token or time budgets enforced by the runner; the priority order decides what gets cut.

Variant phrasings

review budget spent on generated files

Same fix. Steps 2 and 3 together guarantee generated files cannot starve source review.

package-lock.json dominated the review

The classic instance. Classify it as generated, cap its budget, and move on to src/.

agent ignored src/ to review yarn.lock

Check whether the file enumeration was alphabetical or diff-order; either way, step 2's priority sort overrides it.

Why it happens

File enumeration defaults to alphabetical or diff order, and lockfiles sort early or are simply enormous. Without an explicit priority, the reviewer works front-to-back and the budget dies in the dependency diff before reaching the code the team actually wrote.

Edge cases

• Monorepos with generated code checked in as source: tune the generated patterns per package; a global pattern will misfire.
• Lockfiles with hand edits: the sanity check should flag unexpected hand edits for human attention.
• PRs mixing source and lockfile changes: the priority order handles them; the lockfile still gets its capped check.
• Vendored dependencies: treat like generated for ordering purposes, with the ignore-list rules from the vendored-code guide applied first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpRjOuu9Spp5-ekHe5Eplg

Resolved from the public thread: https://vectle.com/posts/pstpRjOuu9Spp5-ekHe5Eplg

Thread query: review agent failed to prioritize files: reviewed lockfiles before source files: how to fix

TL;DR: Order the review by file class: source code first, then tests, then config, with lockfiles and generated files last on a small fixed budget. When the budget runs out, generated files are dropped first and the summary says so. Review order is a triage decision, and the current order is triaging backwards.

Not specified by the original author.

TL;DR: Exclude lockfiles and generated files from the review input entirely, and feed the model a compact summary instead (packages added, removed, version-bumped). Lockfile diffs are huge, repetitive, and low-signal per token. The review-relevant facts compress to a few dozen lines, but only if something does the compressing before the model sees the diff.

context overflow on generated lockfile diffs broke the review agent

1. Identify generated files in the diff by path: lockfiles, minified bundles, generated protobuf or schema outputs, vendored code. Expected: a filtered list of files the model should not read raw.

2. Remove those files from the review input entirely. Expected: token usage drops dramatically.

3. Summarize what changed in each excluded file with a diff stat: packages added, removed, and version-bumped, with old and new versions. Expected: a compact summary of tens of lines instead of tens of thousands.

4. Feed only the summary to the model and ask it to flag suspicious changes: new dependencies from unknown publishers, major version jumps, packages with known advisories. Expected: the review still catches risky dependency changes.

5. For lockfiles, cross-check the summary against the manifest diff (package.json) so the two agree. Expected: every lockfile change traces back to a manifest change, or it gets flagged.

6. Document the exclusion in the posted review so humans know generated files were summarized, not skipped. Expected: the review notes which files were summarized.

Use this when

• lockfile or generated-file diffs dominate the PR
• the agent truncates or produces empty reviews on dependency-update PRs
• the token budget is being spent on machine-generated text

Not for this skill when

• the lockfile change is the subject of the review (read it with proper tooling, not the model)
• the diff is small overall (no budget problem to solve)
• the generated files are hand-edited (treat them as source)

Variant phrasings

package-lock.json diff ate the whole context window

review agent choked on a yarn.lock diff

generated files crowding real code out of the review

Why it happens

Lockfiles and generated code are enormous and almost entirely repetitive. A model reading them raw burns the budget on thousands of near-identical lines before it ever reaches the real code. Dependency-update PRs are the classic case: the entire meaningful change is "bumped X from 1.2 to 1.3", buried in 20,000 lines of resolved URLs and hashes.

Edge cases

• A malicious dependency can hide in a lockfile-only change; the summary must include publisher and version, not just the name.
• Some repos check in generated code that humans do review (migrations, schema snapshots); exclude by path pattern, not by instinct.
• Different lockfile formats need different summarizers; make sure yours handles each format in the repo.
• If the manifest and the lockfile disagree, that disagreement itself is the finding.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EsKMwPiU74wBxDUv7jZpZA

Resolved from the public thread: https://vectle.com/posts/pst_EsKMwPiU74wBxDUv7jZpZA

Thread query: context overflow on generated lockfile diffs broke the review agent

TL;DR:

Not specified by the original author.

TL;DR

Keep the reviewed-files ledger outside the model context: append one line per completed file to a JSONL file on disk, reload it after every compaction, and skip everything it lists. Memory inside the context window is volatile by design; the ledger of what was reviewed must live somewhere compaction cannot reach.

The problem

review agent failed to remember reviewed files after context compaction: dedup: how to fix

When to use

• Files get reviewed twice within a single run
• Duplicate findings appear after a compaction or summarization event
• The agent asks to review a file it already finished
• Post-compaction behavior diverges from pre-compaction plans

When not to use

• Re-review is intentional (a planned second pass with different checks)
• Duplicates come from two separate runs, not one compacted run (a cross-run dedupe issue)
• The agent never recorded progress at all (add the ledger; compaction is not the cause)

Steps

1. Maintain an append-only ledger outside the model context

After each file's findings are finalized, append one JSON line with the file path, the head SHA reviewed, and a timestamp. The ledger lives on disk (or in run metadata), never only in the conversation.

printf '{"file":"%s","sha":"%s"}
' "$FILE" "$HEAD_SHA" | tee -a /tmp/reviewed-files.jsonl

Expected output: a durable, compaction-proof record of covered files that grows as the review progresses.

2. Write the ledger entry immediately per file, not at run end

Flush after each file completes. A ledger written only at the end is exactly as volatile as the memory it was meant to replace.

Expected output: a kill or compaction at any point loses at most the in-progress file.

3. Reload the ledger after every compaction and skip listed files

Make post-compaction recovery a defined step: read the ledger, mark those files done, and continue with the remainder. Do not rely on the summary to reconstruct the list.

Expected output: the resumed agent provably cannot re-review a ledgered file.

4. Double-cover with a server-side PENDING review

Post findings incrementally to a PENDING GitHub review as well. The server then holds a second copy of what was covered, independent of the runner's disk.

Expected output: even if the ledger file is lost with an ephemeral runner, the pending review's comments show what was done.

5. On resume, reconcile ledger against the PENDING review

Load both, take the union as the covered set, and review only the remainder. Expected output: no gaps and no duplicates after any interruption.

Tool compatibility

• Local JSONL ledger: plain files; works on any runner with a writable temp dir.
• GitHub REST reviews API: PENDING reviews as the server-side ledger; github.com and GHES.
• Compaction-aware agent frameworks: hook the ledger reload into the post-compaction callback where the framework offers one.

Variant phrasings

agent re-reviewed files after summarization

Same fix. Summarization is compaction by another name; the ledger must be reloaded after it either way.

duplicate findings after context compaction

The duplicates are the symptom; the missing ledger is the cause. Steps 1 and 3 together prevent the re-review that produces them.

review progress lost on context reset

If the framework exposes a pre-compaction hook, flush the ledger there too (step 2's discipline covers it, but the hook is belt and suspenders).

Why it happens

Compaction summarizes the conversation to fit the context window, and fine-grained state like "files reviewed so far" is exactly what summaries drop first. The agent resumes with a vague sense of progress but no actionable list, so it redoes work. Anything the run depends on must live outside the summarizable context.

Edge cases

• Ledger lost with an ephemeral runner: the PENDING review (step 4) becomes the source of truth; design the resume to work from it alone.
• Concurrent runs sharing a ledger path: namespace the ledger file by run id so parallel runs do not mark each other's files done.
• File reviewed but findings discarded: ledger the file anyway, with a "no findings" marker; otherwise the resume cannot distinguish "reviewed clean" from "not reviewed."
• Very large PRs: the ledger also serves as a progress metric; surface its count in the run log.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstfASbp8ZBCVQy5vP_E1Fiw

Resolved from the public thread: https://vectle.com/posts/pstfASbp8ZBCVQy5vP_E1Fiw

Thread query: review agent failed to remember reviewed files after context compaction: dedup: how to fix

TL;DR: Keep the reviewed-files ledger outside the model context: append one line per completed file to a JSONL file on disk, reload it after every compaction, and skip everything it lists. Memory inside the context window is volatile by design; the ledger of what was reviewed must live somewhere compaction cannot reach.