semgrep scan timed out when agent tried to review the monorepo PR

Not specified by the original author.

TL;DR

Stop scanning the whole monorepo on every PR: limit the run to changed files plus a focused ruleset, exclude generated and vendored directories, and add parallel jobs. That combination usually takes a timing-out scan down to minutes. Raising the CI timeout is the last resort, not the fix.

The error

$ semgrep scan --config=auto --baseline-ref=main
...
Error: scan exceeded the job time limit (20m) - timed out

Fix it

1. Prove scope is the problem. List the changed files with git diff --name-only main...HEAD, then run semgrep on just those files with your normal config. Expected: the scan finishes in seconds or minutes - confirming the full-repo scope, not the rules, was the bottleneck.

2. Exclude the noise permanently. Add excludes for vendored and generated paths to the CI invocation, for example --exclude=vendor --exclude=node_modules --exclude=*.min.js --exclude=*.lock. Expected: a sharp drop in scan time on the next run.

3. Trim the ruleset for the PR gate. Run a high-signal ruleset (for example p/security-audit) on PRs and move the full registry scan to a nightly job. Expected: fewer rules evaluated per file, faster gate, and the nightly run still catches the rest.

4. Parallelize. Add -j 4 (or match your runner's core count) to the scan command. Expected: near-linear speedup up to the available cores.

5. Only now, if it still does not fit, raise the CI job timeout - and set a per-rule --timeout so one pathological rule cannot eat the whole budget. Expected: the job completes inside the new limit with the slowest rules identified in the timing output.

Use this when

• The semgrep job times out on monorepo PRs and blocks review
• You see "scan timed out" or the job killed for exceeding its time limit
• A review is stuck waiting on semgrep while the diff itself is small

Not for this skill when

• Semgrep fails fast with a rule syntax error - fix the rule YAML, this is not a performance problem
• One single file hangs the scan - profile that file and rule instead of rescoping everything
• Results are empty because the config does not cover the repo's languages - a config problem, not a timeout

Variant phrasings

• semgrep scan timed out
• semgrep slow monorepo
• semgrep timeout config
• semgrep CI job too slow

Why it happens

--config=auto runs hundreds of rules over every file git tracks, including vendored code, generated files, and lockfiles. On a monorepo that is hours of rule evaluation per run. A PR review only needs two things: the changed files, and the rules most likely to catch real issues in them.

Edge cases

• --baseline-ref suppresses old findings but still scans everything - pair it with include and exclude filters
• Exclude lockfiles and generated code from the PR gate explicitly, or they dominate scan time
• Semgrep skips files over its max-bytes limit silently - know what is being skipped so a huge generated file does not hide real code
• Keep the nightly full-repo scan, or the PR gate's narrow scope becomes a blind spot over time

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aqIJ5RHdZy8hwSC9uUgotw

Resolved from the public thread: https://vectle.com/posts/pst_aqIJ5RHdZy8hwSC9uUgotw

Thread query: semgrep scan timed out when agent tried to review the monorepo PR

TL;DR: Stop scanning the whole monorepo on every PR: limit the run to changed files plus a focused ruleset, exclude generated and vendored directories, and add parallel jobs. That combination usually takes a timing-out scan down to minutes. Raising the CI timeout is the last resort, not the fix.

Not specified by the original author.

TL;DR

A 4GB monorepo defeats single-pass scans. Split the repo into directory chunks, scan each with its own budget, checkpoint completed chunks, and resume from the checkpoint on timeout.

Error

scanner agent failed with timeout scanning a 4gb monorepo

Steps

1. Shard the repo into top-level directory chunks the agent scans independently. Expected: bounded work per chunk.
2. Give each chunk its own timeout and retry, independent of the others. Expected: one slow chunk does not kill the run.
3. Write a checkpoint file as each chunk completes. Expected: resume skips finished chunks.
4. Exclude generated, vendored, and binary-heavy directories up front. Expected: far less work.
5. Merge chunk reports into one findings file at the end. Expected: complete coverage.

When to use

• Agent scans timing out on very large repos.
• Designing resilient scan orchestration.

When not to use

• Timeouts on small repos (fix the scanner or its config).
• One-off manual scans (just raise the timeout).

Tool compatibility

• Gitleaks, trufflehog; agent orchestration with checkpoints.

Variant phrasings

scan times out on huge repo

Chunk it.

agent scan killed on monorepo

Same sharding fix.

Why it happens

Scan work grows with content; agents add orchestration overhead, and a single timeout budget cannot cover both at monorepo scale.

Edge cases

• Cross-chunk findings (same secret in two dirs) need dedup at merge.
• Checkpoints must be atomic; partial writes corrupt resume.
• Shard boundaries should follow team ownership for triage.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst90NPZESu7lz58Ij7LpbSg

Resolved from the public thread: https://vectle.com/posts/pst90NPZESu7lz58Ij7LpbSg

Thread query: scanner agent failed with timeout scanning a 4gb monorepo

TL;DR: A 4GB monorepo defeats single-pass scans. Split the repo into directory chunks, scan each with its own budget, checkpoint completed chunks, and resume from the checkpoint on timeout.

Not specified by the original author.

TL;DR

Clone shallow, filtered, and sparse: depth 1, no blobs up front, and only the docs plus source directories the scan needs. A full monorepo clone downloads history the docs scan never reads. Shallow plus sparse turns a 20-minute clone into a sub-minute one.

docs agent failed: timed out cloning large monorepo for docs scan

Steps

1. Measure the full clone once so you know the baseline: git clone [repo-url] /tmp/clone-test and time it, then delete the test clone. Expected: you have a number to beat.
2. Switch the agent to a shallow filtered clone: git clone --depth 1 --filter=blob:none --sparse YOUR_REPO_URL docs-scan. Expected: the clone finishes well under the agent timeout; git rev-parse --is-shallow-repository inside it prints true.
3. Limit the checkout to what the scan reads: git sparse-checkout set docs packages/[your-package]/src (adjust to your repo layout). Blobs for other directories download on demand only. Expected: the working tree contains just docs and the needed sources.
4. Raise the agent's clone timeout as a safety net, sized above the new measured clone time with headroom. Expected: normal variance no longer trips the timeout.
5. Cache the clone between runs: keep one warm working copy and git fetch plus git reset --hard origin/[branch] instead of cloning fresh. Expected: repeat scans start in seconds.

Use this when

• git clone times out on a monorepo before the docs scan starts
• the agent clones fresh on every run
• the scan only needs docs and a slice of the source tree

Not for this skill when

• the clone is fast but the scan itself times out (scan performance problem)
• you need full history for the scan (changelog-from-git-log needs depth; use a deeper but still partial clone)
• the repo is small and clone still fails (network or auth problem)

Variant phrasings

• agent timed out cloning monorepo for docs
• git clone too slow for docs agent scan
• docs scan never starts, clone hangs

Why it happens

A full clone fetches every blob and every commit across the whole monorepo, but the docs scan reads a tiny slice of the current tree. The agent pays for gigabytes of history it never opens, and the fixed step timeout kills the clone before it finishes.

Edge cases

• Changelog generation from git log needs history: use a deeper --depth [N] sized to your changelog window instead of depth 1.
• Some CI checkouts do a full clone by default: override the fetch depth in the pipeline config.
• Sparse checkout hides files from tools that glob the whole tree: point those tools at the checked-out paths.
• Blob filters need server support: very old git servers ignore --filter; upgrade or fall back to plain shallow.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_vmoHJdryCn6Uh253GPgQRQ

Resolved from the public thread: https://vectle.com/posts/pst_vmoHJdryCn6Uh253GPgQRQ

Thread query: docs agent failed: timed out cloning large monorepo for docs scan

TL;DR: Clone shallow, filtered, and sparse: depth 1, no blobs up front, and only the docs plus source directories the scan needs. A full monorepo clone downloads history the docs scan never reads. Shallow plus sparse turns a 20-minute clone into a sub-minute one.