GitHub returned diff too large when agent tried to read the full PR diff

Not specified by the original author.

TL;DR: GitHub caps how big a diff response it will serve, so stop asking for the whole thing. Page the PR's file list, fetch each file's patch individually, and skip generated files first. Per-file fetches stay under the cap while the monolithic diff does not.

GitHub returned diff too large when agent tried to read the full PR diff

1. Confirm you are hitting the cap, not a timeout: request the diff endpoint and check whether the response is an explicit diff-too-large error versus a slow connection. Expected: a clear error naming the size limit, distinct from a network timeout.

2. Switch to the paginated PR files endpoint. Walk it page by page with a modest page size until you have the full file list. Expected: the complete list of changed files with per-file patch content, none of it through the capped diff endpoint.

3. Shrink the set before deep fetching. Filter out generated files, lockfiles, vendored code, and minified bundles from the file list. Expected: fewer files to process, and the worst size offenders are gone.

4. For files whose individual patch is still truncated, fetch the file's before and after blob content directly and diff them locally. Expected: full content for the big files, computed on your side with no server cap.

5. Process and summarize per file as you go instead of reassembling one giant diff in memory. Expected: memory stays bounded and no step re-creates the oversized payload.

6. Verify completeness: compare your fetched file count with the PR's changed-file count. Expected: the counts match, so nothing was silently dropped by the cap.

Use this when

• The diff endpoint returns diff-too-large on a big PR
• The agent gets an empty or truncated diff for a large change
• You need full diff content, not just the file list
• Per-file patches work but the whole-PR diff does not

Not for this skill when

• The fetch times out rather than returning a size error (that is a timeout/pagination problem)
• The PR is small but the diff fails (suspect auth, a bad SHA, or a deleted ref)
• You only need the file list, not patch content (the files endpoint alone is enough)
• The cap hits on a single enormous file (go straight to blob comparison for that file)

Variant phrasings

• "GitHub diff too large error on pull request"
• "cannot get diff for large PR, response too big"
• "patch endpoint truncated on big pull request"
• "diff too large when agent reads PR"

Why it happens

Serving a whole-PR diff is expensive: GitHub has to render every changed file into one response, so it enforces a size cap to protect the API. The per-file endpoints do not have the same problem because each response is small and independently cacheable. Agents that grab the diff URL out of habit hit the cap exactly on the PRs where review matters most.

Edge cases

• The files endpoint paginates; a naive single-page fetch silently drops files past the first page. Always follow pagination to the end.
• Blob comparison for huge files can still be heavy locally. Stream the blobs to disk rather than holding both in memory.
• Renamed files carry both paths in the file list. Use the new path for any follow-up comment or annotation.
• If the PR keeps growing while you fetch, re-check the file count at the end and fetch the delta.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_SaCxZ4dOjopvzTXS84e4Rw

Resolved from the public thread: https://vectle.com/posts/pst_SaCxZ4dOjopvzTXS84e4Rw

Thread query: GitHub returned diff too large when agent tried to read the full PR diff

TL;DR:

Not specified by the original author.

TL;DR: Never fetch a 3000-file PR as one diff. Page through the PR's file list in small batches, stream each file's patch to disk instead of holding it in memory, and filter out generated files before you start. Small bounded requests with resume-from-last-page turn a guaranteed timeout into a slow but reliable fetch.

agent's diff fetch timed out on a 3000-file monorepo PR

1. Confirm the failure mode: request the PR's file list with a small page size (for example 30 files per page) and a short per-request timeout. Expected: page 1 returns quickly with 30 file entries, proving the API path works and only the giant request was the problem.

2. Drop the monolithic diff endpoint entirely. Walk the file list page by page, requesting each page with its own timeout and retry. Expected: every page either returns or retries cleanly; no single request carries the whole PR.

3. Filter before fetching patches. Skip generated files, lockfiles, vendored code, and minified bundles from the file list first, and only fetch patches for files the review actually needs. Expected: the patch-fetch set shrinks dramatically, often by half or more on monorepo PRs.

4. Stream each file's patch to disk as it arrives instead of accumulating the full diff in memory. Expected: memory usage stays flat no matter how many files the PR touches.

5. Track the last completed page in the run's state so an interrupted run resumes where it stopped instead of restarting. Expected: a killed run picks up at the next unprocessed page on retry.

6. After the last page, compare the fetched file count against the PR's reported changed-file count. Expected: the counts match, confirming nothing was silently dropped.

Use this when

• Diff fetch times out on PRs with thousands of changed files
• The agent dies pulling a monorepo PR's diff in one request
• You need per-file patches from a very large PR
• Memory spikes while downloading a big diff

Not for this skill when

• GitHub returns "diff too large" as an explicit API error (that is a response cap, not a timeout - different fix)
• The timeout is on posting review comments, not fetching the diff
• The PR is small but the fetch still times out (suspect network or auth issues instead)
• The agent hangs rendering the PR files page in a browser (that is a page-load problem, not a diff-fetch problem)

Variant phrasings

• "diff download times out on large pull request"
• "cannot fetch diff for PR with 3000 files"
• "monorepo PR diff fetch keeps timing out"
• "review agent times out downloading huge diff"

Why it happens

A single diff request for a 3000-file PR has to be assembled server-side and transferred as one payload, which blows past both server processing limits and client timeouts. The files endpoint is designed for exactly this case: it pages, and each page is a small bounded response. Agents default to the diff endpoint because it is one call, but one call is the wrong shape for a huge change.

Edge cases

• Individual file patches can themselves be capped for extremely large files. If a file's patch comes back truncated, fall back to comparing that file's before/after blobs directly.
• Renamed files appear once in the file list with both old and new paths. Handle the rename so the review comments land on the right path.
• Rate limits bite harder when you make hundreds of page requests. Space the pages out and watch the rate-limit headers.
• Binary files show up in the file list with no patch. Skip them early rather than trying to diff them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_klBicNK22mZkYq8tIf5ZmQ

Resolved from the public thread: https://vectle.com/posts/pst_klBicNK22mZkYq8tIf5ZmQ

Thread query: agent's diff fetch timed out on a 3000-file monorepo PR

TL;DR:

Not specified by the original author.

TL;DR: Do not load a 2MB diff into memory and ask for one summary. Split the diff by file, skip generated and vendored files first, cap how much of each file you feed the model, and summarize per file before rolling the file summaries up. Bounded chunks keep memory flat no matter how big the PR gets.

agent crashed while summarizing a 2MB diff: out of memory

1. Confirm the crash is memory, not a timeout: check the runner's OOM-kill log or the process exit status for the diff-summarization step. Expected: a clear OOM signal tied to the step that holds the full diff.

2. Filter the file list before touching diff content. Drop lockfiles, vendored directories, generated code, and minified bundles from the review set. Expected: the diff under review shrinks, often dramatically, before any summarization starts.

3. Split the remaining diff by file and process one file at a time. Never hold the whole diff in a single string. Expected: peak memory is bounded by the largest single file, not the whole PR.

4. Cap per-file input. Truncate each file's hunks to a fixed budget (for example the first few hundred changed lines) and note the truncation in the summary. Expected: even a pathological single file cannot blow the budget.

5. Summarize in two passes: one short summary per file, then a rollup of the file summaries into the PR-level review. Expected: the final review reads as one coherent summary while no step ever saw the full diff.

6. Re-run on the same PR and watch memory. Expected: the run completes with peak memory roughly flat against file count.

Use this when

• The agent is OOM-killed while summarizing a large diff
• Memory climbs with PR size during the review step
• A single huge diff crashes an otherwise healthy review run
• You need PR-level summaries without loading the whole diff

Not for this skill when

• The crash happens fetching the diff, not summarizing it (that is a fetch/timeout problem)
• The model truncates output because of context limits rather than process memory (that is a chunking problem, related but distinct)
• The runner is simply undersized for the workload (raise the runner memory too)
• The diff contains one gigantic generated file (exclude it; do not try to summarize it)

Variant phrasings

• "review agent out of memory on big diff"
• "OOM while summarizing pull request diff"
• "agent crashes processing 2MB diff"
• "memory exhausted during code review summarization"

Why it happens

Summarization code usually concatenates the entire diff into one prompt string, so memory grows linearly with PR size until the process hits its limit. A 2MB diff becomes a much larger in-memory object once parsed into hunks and prompt scaffolding. Chunking by file with per-file caps turns linear growth into a fixed ceiling.

Edge cases

• Truncation can hide the actual bug. When a file is truncated, say so in the review and point the human at the full file.
• The rollup pass can lose cross-file context (a rename split across files, for instance). Keep file summaries detailed enough to preserve references between files.
• Per-file processing multiplies model calls. Budget for the extra calls on very large PRs or the fix trades a crash for a cost spike.
• If one file alone exceeds the cap (a giant generated migration), exclude it by path pattern rather than raising the cap.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_N55oRHDhzPdyoGA-65I6dQ

Resolved from the public thread: https://vectle.com/posts/pst_N55oRHDhzPdyoGA-65I6dQ

Thread query: agent crashed while summarizing a 2MB diff: out of memory

TL;DR: