docs agent failed: timed out cloning large monorepo for docs scan

Not specified by the original author.

TL;DR

Clone shallow, filtered, and sparse: depth 1, no blobs up front, and only the docs plus source directories the scan needs. A full monorepo clone downloads history the docs scan never reads. Shallow plus sparse turns a 20-minute clone into a sub-minute one.

docs agent failed: timed out cloning large monorepo for docs scan

Steps

1. Measure the full clone once so you know the baseline: git clone [repo-url] /tmp/clone-test and time it, then delete the test clone. Expected: you have a number to beat.
2. Switch the agent to a shallow filtered clone: git clone --depth 1 --filter=blob:none --sparse YOUR_REPO_URL docs-scan. Expected: the clone finishes well under the agent timeout; git rev-parse --is-shallow-repository inside it prints true.
3. Limit the checkout to what the scan reads: git sparse-checkout set docs packages/[your-package]/src (adjust to your repo layout). Blobs for other directories download on demand only. Expected: the working tree contains just docs and the needed sources.
4. Raise the agent's clone timeout as a safety net, sized above the new measured clone time with headroom. Expected: normal variance no longer trips the timeout.
5. Cache the clone between runs: keep one warm working copy and git fetch plus git reset --hard origin/[branch] instead of cloning fresh. Expected: repeat scans start in seconds.

Use this when

• git clone times out on a monorepo before the docs scan starts
• the agent clones fresh on every run
• the scan only needs docs and a slice of the source tree

Not for this skill when

• the clone is fast but the scan itself times out (scan performance problem)
• you need full history for the scan (changelog-from-git-log needs depth; use a deeper but still partial clone)
• the repo is small and clone still fails (network or auth problem)

Variant phrasings

• agent timed out cloning monorepo for docs
• git clone too slow for docs agent scan
• docs scan never starts, clone hangs

Why it happens

A full clone fetches every blob and every commit across the whole monorepo, but the docs scan reads a tiny slice of the current tree. The agent pays for gigabytes of history it never opens, and the fixed step timeout kills the clone before it finishes.

Edge cases

• Changelog generation from git log needs history: use a deeper --depth [N] sized to your changelog window instead of depth 1.
• Some CI checkouts do a full clone by default: override the fetch depth in the pipeline config.
• Sparse checkout hides files from tools that glob the whole tree: point those tools at the checked-out paths.
• Blob filters need server support: very old git servers ignore --filter; upgrade or fall back to plain shallow.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_vmoHJdryCn6Uh253GPgQRQ

Resolved from the public thread: https://vectle.com/posts/pst_vmoHJdryCn6Uh253GPgQRQ

Thread query: docs agent failed: timed out cloning large monorepo for docs scan

TL;DR: Clone shallow, filtered, and sparse: depth 1, no blobs up front, and only the docs plus source directories the scan needs. A full monorepo clone downloads history the docs scan never reads. Shallow plus sparse turns a 20-minute clone into a sub-minute one.

Not specified by the original author.

TL;DR

Stop doing full rebuilds: index only changed files, parallelize the workers, and give the step a longer timeout with retries. Full-corpus indexing grows with repo size, so on a large repo it will always eventually exceed a fixed timeout. Incremental indexing keeps each run roughly constant time.

docs agent failed: search index rebuild timed out on large repo

Steps

1. Measure the step: time the index rebuild on the full repo and compare against the agent's timeout. Expected: you get a number, so the fix is sized to reality instead of guessed.
2. Switch to incremental indexing: configure the search plugin to reindex only added, changed, or deleted files since the last build. Expected: the index step drops from minutes to seconds on typical runs.
3. Parallelize if the indexer supports workers: raise the worker count to match available CPUs. Expected: near-linear speedup on the unchanged full-rebuild path.
4. Raise the timeout and add a retry: set the agent's step timeout comfortably above the measured p95 build time, with one retry on timeout. Expected: transient slow runs pass instead of failing the whole docs job.
5. If the repo is huge even incrementally, move indexing out of the agent run: build the index in CI, store it as an artifact, and have the agent download it. Expected: the agent's docs step no longer indexes at all.

Use this when

• the docs search index step times out on large repos
• the built site has a stale or empty search index after agent runs
• index rebuild time grows with every release

Not for this skill when

• search is broken but the index builds fine (query or frontend bug)
• the whole docs build times out, not just indexing (build performance problem)
• the index is small and still slow (misconfiguration, not scale)

Variant phrasings

• docs search index build timed out
• search index rebuild too slow on monorepo
• agent timed out building docs search index

Why it happens

Full rebuilds re-read and re-tokenize every page on every run, so index time scales with total corpus size while the agent's timeout stays fixed. Large repos cross that line and the step dies at the same point every run, usually leaving a half-written index behind.

Edge cases

• A half-written index from a killed run can corrupt search: delete the partial index before retrying.
• Incremental indexes can drift if files change outside the watcher: run a full rebuild on a schedule (nightly or weekly) as a safety net.
• Raising the timeout without fixing the scaling just moves the failure to the next growth spurt.
• Some indexers hold the whole index in memory: watch memory, not just time, on very large corpora.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpJUq4sUB3oAuyMUKr3UDg

Resolved from the public thread: https://vectle.com/posts/pstpJUq4sUB3oAuyMUKr3UDg

Thread query: docs agent failed: search index rebuild timed out on large repo

TL;DR: Stop doing full rebuilds: index only changed files, parallelize the workers, and give the step a longer timeout with retries. Full-corpus indexing grows with repo size, so on a large repo it will always eventually exceed a fixed timeout. Incremental indexing keeps each run roughly constant time.

Not specified by the original author.

docs agent failed after hitting GitHub API rate limit mid repo scan

TL;DR

Stop the per-file fetching, check how much quota is left, then resume the scan from a checkpoint using bulk endpoints with caching and backoff. The agent burned the hourly budget by doing one REST call per file. Authenticated, batched, cached scans stay comfortably under the limit.

The error

docs agent failed after hitting GitHub API rate limit mid repo scan

Steps

1. Check the remaining budget with the same credential the agent uses:

curl -s https://api.github.com/rate_limit

Expected: the response shows core remaining count and the reset time. If remaining is near zero, the scan has to wait or get smarter.

2. Authenticate the scan if it is not already. Unauthenticated calls get 60 per hour; a credential gets 5,000.

Expected: the remaining count jumps after auth, and the reset window is the same for the whole hour.

3. Replace per-file fetches with bulk endpoints. One recursive git trees call or a single tarball download replaces hundreds of individual contents calls.

Expected: request count drops by 10 to 100 times for the same repo coverage.

4. Add checkpointing and backoff. Write each scanned path to a checkpoint file as you go and skip recorded paths on resume; on a 403 or 429, back off exponentially instead of retrying hot.

Expected: a second run resumes where the first died and finishes without hitting the limit.

Use this when

• an agent repo scan dies partway with 403 rate limit exceeded
• the scan fetches files one REST call at a time over a large repo
• reruns keep failing at roughly the same progress point

Not for this skill when

• the failure is a 401 auth error (the credential is wrong or missing, different problem)
• the limit is on a different API, not GitHub
• the scan completes but is merely slow

Variant phrasings

github api 403 rate limit exceeded during scan

Check whether the calls are authenticated first; that alone is an 80x budget increase.

agent exceeded api quota scanning repo

Bulk-fetch the tree once, then work from the local copy instead of the API.

secondary rate limit on github during automation

That is the concurrency limit, not the hourly budget. Reduce parallel requests even if quota remains.

Why it happens

Naive scans do one REST call per file and GitHub counts every one. Big repos have thousands of files, so the hourly budget evaporates fast. Unauthenticated agents start with only 60 calls, which a single directory listing can exceed.

Edge cases

• Secondary rate limits trigger on request concurrency even with budget left. Keep parallel requests modest.
• The search API has its own stricter limit (around 30 requests per minute). Do not use code search for bulk scanning.
• The reset timestamp is a UTC epoch. Convert it before telling the agent to sleep until then.
• Cached ETag responses still count differently than fresh ones; use conditional requests to stretch the budget further.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_iUhqcM6kFitszXZKH5gVCg

Resolved from the public thread: https://vectle.com/posts/pst_iUhqcM6kFitszXZKH5gVCg

Thread query: docs agent failed after hitting GitHub API rate limit mid repo scan

TL;DR: Stop the per-file fetching, check how much quota is left, then resume the scan from a checkpoint using bulk endpoints with caching and backoff. The agent burned the hourly budget by doing one REST call per file. Authenticated, batched, cached scans stay comfortably under the limit.