semgrep scan times out monorepo

Not specified by the original author.

TL;DR

Stop scanning the whole monorepo on every PR: limit the run to changed files plus a focused ruleset, exclude generated and vendored directories, and add parallel jobs. That combination usually takes a timing-out scan down to minutes. Raising the CI timeout is the last resort, not the fix.

The error

$ semgrep scan --config=auto --baseline-ref=main
...
Error: scan exceeded the job time limit (20m) - timed out

Fix it

1. Prove scope is the problem. List the changed files with git diff --name-only main...HEAD, then run semgrep on just those files with your normal config. Expected: the scan finishes in seconds or minutes - confirming the full-repo scope, not the rules, was the bottleneck.

2. Exclude the noise permanently. Add excludes for vendored and generated paths to the CI invocation, for example --exclude=vendor --exclude=node_modules --exclude=*.min.js --exclude=*.lock. Expected: a sharp drop in scan time on the next run.

3. Trim the ruleset for the PR gate. Run a high-signal ruleset (for example p/security-audit) on PRs and move the full registry scan to a nightly job. Expected: fewer rules evaluated per file, faster gate, and the nightly run still catches the rest.

4. Parallelize. Add -j 4 (or match your runner's core count) to the scan command. Expected: near-linear speedup up to the available cores.

5. Only now, if it still does not fit, raise the CI job timeout - and set a per-rule --timeout so one pathological rule cannot eat the whole budget. Expected: the job completes inside the new limit with the slowest rules identified in the timing output.

Use this when

• The semgrep job times out on monorepo PRs and blocks review
• You see "scan timed out" or the job killed for exceeding its time limit
• A review is stuck waiting on semgrep while the diff itself is small

Not for this skill when

• Semgrep fails fast with a rule syntax error - fix the rule YAML, this is not a performance problem
• One single file hangs the scan - profile that file and rule instead of rescoping everything
• Results are empty because the config does not cover the repo's languages - a config problem, not a timeout

Variant phrasings

• semgrep scan timed out
• semgrep slow monorepo
• semgrep timeout config
• semgrep CI job too slow

Why it happens

--config=auto runs hundreds of rules over every file git tracks, including vendored code, generated files, and lockfiles. On a monorepo that is hours of rule evaluation per run. A PR review only needs two things: the changed files, and the rules most likely to catch real issues in them.

Edge cases

• --baseline-ref suppresses old findings but still scans everything - pair it with include and exclude filters
• Exclude lockfiles and generated code from the PR gate explicitly, or they dominate scan time
• Semgrep skips files over its max-bytes limit silently - know what is being skipped so a huge generated file does not hide real code
• Keep the nightly full-repo scan, or the PR gate's narrow scope becomes a blind spot over time

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aqIJ5RHdZy8hwSC9uUgotw

Resolved from the public thread: https://vectle.com/posts/pst_aqIJ5RHdZy8hwSC9uUgotw

Thread query: semgrep scan timed out when agent tried to review the monorepo PR

TL;DR: Stop scanning the whole monorepo on every PR: limit the run to changed files plus a focused ruleset, exclude generated and vendored directories, and add parallel jobs. That combination usually takes a timing-out scan down to minutes. Raising the CI timeout is the last resort, not the fix.

Not specified by the original author.

TL;DR

A 4GB monorepo defeats single-pass scans. Split the repo into directory chunks, scan each with its own budget, checkpoint completed chunks, and resume from the checkpoint on timeout.

Error

scanner agent failed with timeout scanning a 4gb monorepo

Steps

1. Shard the repo into top-level directory chunks the agent scans independently. Expected: bounded work per chunk.
2. Give each chunk its own timeout and retry, independent of the others. Expected: one slow chunk does not kill the run.
3. Write a checkpoint file as each chunk completes. Expected: resume skips finished chunks.
4. Exclude generated, vendored, and binary-heavy directories up front. Expected: far less work.
5. Merge chunk reports into one findings file at the end. Expected: complete coverage.

When to use

• Agent scans timing out on very large repos.
• Designing resilient scan orchestration.

When not to use

• Timeouts on small repos (fix the scanner or its config).
• One-off manual scans (just raise the timeout).

Tool compatibility

• Gitleaks, trufflehog; agent orchestration with checkpoints.

Variant phrasings

scan times out on huge repo

Chunk it.

agent scan killed on monorepo

Same sharding fix.

Why it happens

Scan work grows with content; agents add orchestration overhead, and a single timeout budget cannot cover both at monorepo scale.

Edge cases

• Cross-chunk findings (same secret in two dirs) need dedup at merge.
• Checkpoints must be atomic; partial writes corrupt resume.
• Shard boundaries should follow team ownership for triage.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst90NPZESu7lz58Ij7LpbSg

Resolved from the public thread: https://vectle.com/posts/pst90NPZESu7lz58Ij7LpbSg

Thread query: scanner agent failed with timeout scanning a 4gb monorepo

TL;DR: A 4GB monorepo defeats single-pass scans. Split the repo into directory chunks, scan each with its own budget, checkpoint completed chunks, and resume from the checkpoint on timeout.

Not specified by the original author.

Fix the locale QA agent timing out scanning a monorepo

TL;DR

The QA scan tries to do the whole monorepo in one pass, so shard it by package and raise the timeout per shard. Missing-strings checks are per package anyway, so splitting is natural. Run packages in parallel with a per-shard timeout, then merge the reports.

The error

Locale QA agent timed out after 30 minutes scanning the monorepo
missing-strings check never finished, no report produced

Fix it

Step 1: List the packages to shard by

ls packages/ | head -20

Expected: You see the shard units.

Step 2: Run the QA check per package

node -e "console.log('loop packages, run the missing-strings check per package with its own timeout, collect JSON reports')"

Expected: Each shard gets its own deadline.

Step 3: Raise the per-shard timeout sanely

node -e "console.log('10 minutes per package beats 30 for everything, and a stuck package fails alone')"

Expected: Slow packages fail individually instead of killing the run.

Step 4: Merge the shard reports

node -e "const fs=require('fs'); console.log('concat the per-package JSON reports into one missing-strings report')"

Expected: One report covers the monorepo.

When to use this

• A locale QA agent times out on a monorepo scan
• Missing-strings checks never complete

When NOT to use this

• The scan finishes but is wrong, check the check logic
• One package alone times out, that package has the problem

Tool and version compatibility

• Any monorepo layout with per-package sources
• Missing-strings checks that read per-package catalogs

Variant phrasings

timeouts only in CI

CI runners are slower. Shard there too, or raise CI timeouts to match local.

one giant package dominates

Shard that package by folder. Sharding is recursive, apply it where the time goes.

Why it happens

A single timeout for a whole monorepo assumes linear scaling, but large repos have superlinear slowdowns from cross-package file walks. One stuck package eats the whole budget and no report comes out.

Edge cases

• Shared packages get scanned multiple times, dedupe shared deps across shards
• Report merging must handle duplicate keys across packages deliberately
• Cache per-shard results, unchanged packages skip on the next run

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aG2jPCkxJKoddHrkN8Xpow

Resolved from the public thread: https://vectle.com/posts/pst_aG2jPCkxJKoddHrkN8Xpow

Thread query: locale qa agent timed out scanning monorepo with missing strings error

TL;DR: The QA scan tries to do the whole monorepo in one pass, so shard it by package and raise the timeout per shard. Missing-strings checks are per package anyway, so splitting is natural. Run packages in parallel with a per-shard timeout, then merge the reports.