review agent failed to remember reviewed files after context compaction: dedup: how to fix

TL;DR

Keep the reviewed-files ledger outside the model context: append one line per completed file to a JSONL file on disk, reload it after every compaction, and skip everything it lists. Memory inside the context window is volatile by design; the ledger of what was reviewed must live somewhere compaction cannot reach.

The problem

review agent failed to remember reviewed files after context compaction: dedup: how to fix

When to use

• Files get reviewed twice within a single run
• Duplicate findings appear after a compaction or summarization event
• The agent asks to review a file it already finished
• Post-compaction behavior diverges from pre-compaction plans

When not to use

• Re-review is intentional (a planned second pass with different checks)
• Duplicates come from two separate runs, not one compacted run (a cross-run dedupe issue)
• The agent never recorded progress at all (add the ledger; compaction is not the cause)

Steps

1. Maintain an append-only ledger outside the model context

After each file's findings are finalized, append one JSON line with the file path, the head SHA reviewed, and a timestamp. The ledger lives on disk (or in run metadata), never only in the conversation.

printf '{"file":"%s","sha":"%s"}
' "$FILE" "$HEAD_SHA" | tee -a /tmp/reviewed-files.jsonl

Expected output: a durable, compaction-proof record of covered files that grows as the review progresses.

2. Write the ledger entry immediately per file, not at run end

Flush after each file completes. A ledger written only at the end is exactly as volatile as the memory it was meant to replace.

Expected output: a kill or compaction at any point loses at most the in-progress file.

3. Reload the ledger after every compaction and skip listed files

Make post-compaction recovery a defined step: read the ledger, mark those files done, and continue with the remainder. Do not rely on the summary to reconstruct the list.

Expected output: the resumed agent provably cannot re-review a ledgered file.

4. Double-cover with a server-side PENDING review

Post findings incrementally to a PENDING GitHub review as well. The server then holds a second copy of what was covered, independent of the runner's disk.

Expected output: even if the ledger file is lost with an ephemeral runner, the pending review's comments show what was done.

5. On resume, reconcile ledger against the PENDING review

Load both, take the union as the covered set, and review only the remainder. Expected output: no gaps and no duplicates after any interruption.

Tool compatibility

• Local JSONL ledger: plain files; works on any runner with a writable temp dir.
• GitHub REST reviews API: PENDING reviews as the server-side ledger; github.com and GHES.
• Compaction-aware agent frameworks: hook the ledger reload into the post-compaction callback where the framework offers one.

Variant phrasings

agent re-reviewed files after summarization

Same fix. Summarization is compaction by another name; the ledger must be reloaded after it either way.

duplicate findings after context compaction

The duplicates are the symptom; the missing ledger is the cause. Steps 1 and 3 together prevent the re-review that produces them.

review progress lost on context reset

If the framework exposes a pre-compaction hook, flush the ledger there too (step 2's discipline covers it, but the hook is belt and suspenders).

Why it happens

Compaction summarizes the conversation to fit the context window, and fine-grained state like "files reviewed so far" is exactly what summaries drop first. The agent resumes with a vague sense of progress but no actionable list, so it redoes work. Anything the run depends on must live outside the summarizable context.

Edge cases

• Ledger lost with an ephemeral runner: the PENDING review (step 4) becomes the source of truth; design the resume to work from it alone.
• Concurrent runs sharing a ledger path: namespace the ledger file by run id so parallel runs do not mark each other's files done.
• File reviewed but findings discarded: ledger the file anyway, with a "no findings" marker; otherwise the resume cannot distinguish "reviewed clean" from "not reviewed."
• Very large PRs: the ledger also serves as a progress metric; surface its count in the run log.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstfASbp8ZBCVQy5vP_E1Fiw