review agent's test run on the PR timed out: 30-minute review budget

Not specified by the original author.

TL;DR

Do not run the full suite in the review path: run only the tests covering the changed files, wrap every invocation in a hard timeout, parallelize, and report honestly when tests were skipped. A review that dies waiting for tests delivers nothing; a review that reports partial test results still delivers the review.

The problem

review agent's test run on the PR timed out: 30-minute review budget blown: how to fix

When to use

• The test step exceeds the review job's timeout
• Review jobs are killed at the time limit during tests
• Slow suites make every review late
• A hung test hangs the entire review

When not to use

• Tests fail fast (a test-quality issue, not a time issue)
• The suite is fast and the timeout came from elsewhere (profile first)
• Policy requires the full suite per PR (then raise the budget; do not pretend it fits)

Steps

1. Run affected tests only, not the whole suite

Map changed files to the tests that cover them (per-language tooling: jest changed-since, pytest test selection, go test on changed packages) and run that subset.

npx jest --changedSince=origin/main --passWithNoTests

Expected output: a test run proportional to the PR's size instead of the repo's size.

2. Wrap every test invocation in a hard timeout

Use the timeout command or the runner's timeout-minutes so a hung test fails fast instead of eating the budget.

timeout 600 npx jest --changedSince=origin/main --passWithNoTests
echo "exit: $?"

Expected output: exit 124 on timeout instead of a hung job; the review step survives to report.

3. Parallelize where the suite allows

Use pytest-xdist, jest workers, or go test parallelism, and shard large suites across runners when the budget is tight.

Expected output: wall-clock test time drops by roughly the parallelism factor on multi-core runners.

4. On budget breach, report partial results honestly

If the targeted run still exceeds the budget, stop, report which tests ran and their result, and mark the rest as not-run. Never let the review die silently with the tests.

Expected output: reviewers see "12/15 test files passed, 3 not run (budget)" instead of a dead job.

5. Move the slow full suite out of the review path

Keep the full suite as a separate scheduled or merge-queue job, not a per-PR review step. Expected output: reviews stay fast; full-suite signal still exists where it belongs.

Tool compatibility

• timeout from coreutils: available on Linux and macOS runners.
• jest --changedSince, pytest with selection plugins, go test on changed packages: per-language affected-test mechanisms.
• CI timeout-minutes: the outer guard on GitHub Actions and equivalents.

Variant phrasings

test suite too slow for PR review

Same fix. Step 1 is the highest-leverage change; most "too slow" suites are running 10x more tests than the PR needs.

review job killed while running tests

Add step 2's hard timeout inside the job so the kill becomes a controlled failure with partial results instead of a dead job.

hung test blocked the code review

Timeouts (step 2) convert hangs into failures; step 4 converts failures into reported partial results. The review itself is never the casualty.

Why it happens

Review pipelines bolt the repo's standard test command onto the review job, and the standard command is tuned for CI thoroughness, not review latency. As the repo grows, the suite outgrows the review budget, and the first symptom is reviews dying inside the test step.

Edge cases

• Integration tests that genuinely need the full budget: move them out of the review path (step 5); do not stretch the review budget to fit them.
• Flaky tests that hang intermittently: timeouts plus retries-with-backoff; track flake rate separately.
• Tests requiring external services: stub or skip in review; run for real in the merge-queue job.
• Zero tests selected: --passWithNoTests (or the equivalent) keeps docs-only PRs green instead of failing on an empty selection.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_76-Zwm-5jZhdudYOpxeoYQ

Resolved from the public thread: https://vectle.com/posts/pst_76-Zwm-5jZhdudYOpxeoYQ

Thread query: review agent's test run on the PR timed out: 30-minute review budget blown: how to fix

TL;DR: Do not run the full suite in the review path: run only the tests covering the changed files, wrap every invocation in a hard timeout, parallelize, and report honestly when tests were skipped. A review that dies waiting for tests delivers nothing; a review that reports partial test results still delivers the review.

Not specified by the original author.

TL;DR

Stop making one API call per file and resume from where the agent stopped. Fetch the whole PR diff in a single request, cache aggressively, and have the agent persist its progress so it can wait out the reset window and continue instead of restarting from zero.

Verbatim error

code review agent hit GitHub API rate limit mid-review of a 40-file PR

Steps

1. Confirm it is the primary rate limit. Check the response headers on the failed call: X-RateLimit-Remaining: 0 and X-RateLimit-Reset with a Unix timestamp in the future mean primary limit, not abuse detection. Expected: you can compute the wait as reset time minus now, in minutes.

2. Cut the call count. Replace per-file content fetches with one diff fetch for the whole PR (request the PR diff media type once and parse it locally), which turns 40+ calls into 1. Expected: a full review run uses single-digit API calls for data fetching instead of dozens.

3. Make the agent resume-aware. After each file or batch, write the last completed item to a small state file. On startup the agent reads it and skips finished work. Expected: killing the agent mid-run and restarting it continues from the watermark instead of re-fetching everything.

4. Add polite backoff. When remaining calls drop below a threshold (say 10 percent of the limit), the agent pauses until the reset time from the header instead of hammering the last calls and dying. Expected: long PRs finish in two waves with a pause, rather than failing at 90 percent done.

Use this when

• A review agent exhausts the GitHub REST rate limit mid-PR.
• You need to reduce API calls in an agent's review loop.
• An agent must survive a rate limit reset and resume cleanly.

Not for this skill when

• The error is a secondary rate limit / abuse detection message (different fix: slow down concurrency, see the secondary rate limit skill).
• Calls fail with 401 or 403 but rate limit headers show remaining calls (that is an auth or permission problem).
• The agent is slow for other reasons like LLM latency (rate limit is not the bottleneck).

Variant phrasings

• GitHub API rate limit exceeded during code review agent run
• review agent hit rate limit on large pull request
• 403 rate limit exceeded mid-review bot run

Why it happens

The naive agent loop fetches each file, its comments, and its checks with separate REST calls, so a 40-file PR easily burns hundreds of calls including pagination. Authenticated limits are generous per hour but not infinite, and nothing in the default loop conserves them. The agent runs out of budget before it runs out of files.

Edge cases

• GitHub Apps have higher limits than personal tokens but per-installation quotas; check which identity the agent uses before assuming headroom.
• Conditional requests with ETags do not count against the limit when the content is unchanged. Cache file contents by SHA and revalidate instead of re-fetching.
• Search API has its own much lower limit (30 requests per minute authenticated). If the agent searches code, that budget dies first.
• Parallel workers multiply consumption. Cap concurrency or give each worker its own token and budget.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VuYwAFgTf9-6ky1R4nYcBg

Resolved from the public thread: https://vectle.com/posts/pst_VuYwAFgTf9-6ky1R4nYcBg

Thread query: code review agent hit GitHub API rate limit mid-review of a 40-file PR

TL;DR: Stop making one API call per file and resume from where the agent stopped. Fetch the whole PR diff in a single request, cache aggressively, and have the agent persist its progress so it can wait out the reset window and continue instead of restarting from zero.

Not specified by the original author.

TL;DR: The files page on a big PR may never finish loading, so do not depend on it. Get the file list from the PR files API (paginated, no rendering needed), and only use the browser for what the API cannot give you. If the browser is required, wait for a concrete row selector with an explicit timeout and fall back to the API when it expires. The review proceeds either way.

review agent hung waiting for the PR files page to finish loading

1. Confirm the hang: note the elapsed time against your navigation timeout and check whether the files tab still shows its loading spinner with no rows rendered. Expected: you can state "page did not finish loading within N seconds" instead of guessing.

2. Stop treating the page as the source of truth for the file list. Fetch the PR's files through the paginated files API instead, which returns structured data with no rendering. Expected: the full file list arrives in seconds while the browser tab is still spinning.

3. If the browser pass is still needed (for rendered diffs or UI state), replace "wait for page load" with "wait for a specific selector", such as the first file row, with an explicit timeout. Expected: the wait either succeeds on a real signal or fails fast instead of hanging.

4. Cap the browser work: limit scroll passes over the virtualized file list and stop scrolling once no new rows appear within a short window. Expected: scrolling terminates even on PRs whose list keeps growing.

5. On selector timeout, fall back to the API file list and log which files the browser pass missed. Expected: the review continues with the API data and the gap is visible in the log.

6. Re-run on the same PR and compare. Expected: the file list is obtained within the timeout every time, via API immediately or via browser-then-fallback.

Use this when

• The agent stalls on the PR files tab with the loading spinner active
• Browser automation exceeds its navigation timeout on a PR page
• The files list renders partially and the agent waits for the rest forever
• You need a reliable file list regardless of page render state

Not for this skill when

• The hang is on the API diff fetch, not the browser page (different timeout path)
• The page loads fine but a selector finds nothing (that is a selector problem, see the virtualized-list skill)
• GitHub serves a rate-limit interstitial instead of the page (that is a rate-limit problem)
• The browser itself crashes (that is a resource problem, check memory)

Variant phrasings

• "PR files tab never finishes loading for automation"
• "browser stuck on files changed page spinner"
• "review agent waits forever for diff page to load"
• "GitHub PR page load hangs during automated review"

Why it happens

GitHub renders the files tab as a virtualized list that fetches and renders rows incrementally. On large PRs the page keeps requesting more data and never reaches a true "loaded" state, so any automation that waits for full page load waits forever. The API has no rendering step, so it does not share the failure mode.

Edge cases

• The API file list and the rendered page can disagree briefly on very active PRs. Trust the API's pagination, not the DOM row count.
• Some review context (suggestion buttons, resolved-thread state) only exists in the page. Scope browser use to those bits and keep the file list on the API.
• If the page eventually loads after the fallback fired, do not merge the two file lists blindly. Dedupe by file path.
• Headless browsers on constrained runners render slower. What loads in 10 seconds on your dev machine can take a minute in CI.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_8rLHBYWmBIc8Kipy7QhR8w

Resolved from the public thread: https://vectle.com/posts/pst_8rLHBYWmBIc8Kipy7QhR8w

Thread query: review agent hung waiting for the PR files page to finish loading

TL;DR: