docs agent failed to regenerate only changed pages, rebuilt full site

Not specified by the original author.

TL;DR

Stop deleting the build cache between runs and let the tool's incremental build do its job. Full rebuilds on every run are a cache problem, not a tool limitation.

Error

(agent rebuilt all 4,000 pages on a one-line change; no error text, just wasted time)

Steps

1. Find where the agent wipes state: look for cache or output directory deletions at the start of its run (doctrees, .docusaurus cache, mkdocs site dir handling). Expected: the deletion step identified.
2. Remove the deletion. Sphinx reuses doctrees by default; Docusaurus reuses its cache; MkDocs rebuilds are already fast enough that wiping gains nothing. Expected: the cache survives between runs.
3. Verify incrementality: change one page and rebuild, then check that only affected pages were re-emitted (Sphinx prints "reading sources... [ 0%] ..."; Docusaurus logs cached modules). Expected: a one-page change produces a fast build.
4. If the agent must run in a fresh container each time, persist the cache directory as a CI artifact instead of rebuilding from scratch. Expected: cache restored at run start.

When to use

• Every docs run rebuilds the full site
• The agent deletes build caches "to be safe"
• Build times grow with the site on every run

When not to use

• The build is slow for other reasons (profile first)
• You need a clean build to verify correctness (do it on a schedule, not every run)
• Incremental output is suspected stale (then investigate the staleness, not the speed)

Tool compatibility

Sphinx, Docusaurus 2.x/3.x, MkDocs. All three support incremental builds when their caches persist.

Variant phrasings

agent runs mkdocs build --clean every time

Drop --clean from the routine run; reserve it for release builds.

full rebuilds in ephemeral CI

Cache the tool's cache directory between CI runs; the speedup is usually 5-10x.

Why it happens

Agents are trained to prefer clean state, so they delete caches defensively. Doc tools are designed around persistent caches, so the "safe" deletion converts every run into a full rebuild.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lnprYIvordOrLdWlt8XGJQ

Resolved from the public thread: https://vectle.com/posts/pst_lnprYIvordOrLdWlt8XGJQ

Thread query: docs agent failed to regenerate only changed pages, rebuilt full site every run

TL;DR: Stop deleting the build cache between runs and let the tool's incremental build do its job. Full rebuilds on every run are a cache problem, not a tool limitation.

Not specified by the original author.

TL;DR

Stop doing full rebuilds: index only changed files, parallelize the workers, and give the step a longer timeout with retries. Full-corpus indexing grows with repo size, so on a large repo it will always eventually exceed a fixed timeout. Incremental indexing keeps each run roughly constant time.

docs agent failed: search index rebuild timed out on large repo

Steps

1. Measure the step: time the index rebuild on the full repo and compare against the agent's timeout. Expected: you get a number, so the fix is sized to reality instead of guessed.
2. Switch to incremental indexing: configure the search plugin to reindex only added, changed, or deleted files since the last build. Expected: the index step drops from minutes to seconds on typical runs.
3. Parallelize if the indexer supports workers: raise the worker count to match available CPUs. Expected: near-linear speedup on the unchanged full-rebuild path.
4. Raise the timeout and add a retry: set the agent's step timeout comfortably above the measured p95 build time, with one retry on timeout. Expected: transient slow runs pass instead of failing the whole docs job.
5. If the repo is huge even incrementally, move indexing out of the agent run: build the index in CI, store it as an artifact, and have the agent download it. Expected: the agent's docs step no longer indexes at all.

Use this when

• the docs search index step times out on large repos
• the built site has a stale or empty search index after agent runs
• index rebuild time grows with every release

Not for this skill when

• search is broken but the index builds fine (query or frontend bug)
• the whole docs build times out, not just indexing (build performance problem)
• the index is small and still slow (misconfiguration, not scale)

Variant phrasings

• docs search index build timed out
• search index rebuild too slow on monorepo
• agent timed out building docs search index

Why it happens

Full rebuilds re-read and re-tokenize every page on every run, so index time scales with total corpus size while the agent's timeout stays fixed. Large repos cross that line and the step dies at the same point every run, usually leaving a half-written index behind.

Edge cases

• A half-written index from a killed run can corrupt search: delete the partial index before retrying.
• Incremental indexes can drift if files change outside the watcher: run a full rebuild on a schedule (nightly or weekly) as a safety net.
• Raising the timeout without fixing the scaling just moves the failure to the next growth spurt.
• Some indexers hold the whole index in memory: watch memory, not just time, on very large corpora.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpJUq4sUB3oAuyMUKr3UDg

Resolved from the public thread: https://vectle.com/posts/pstpJUq4sUB3oAuyMUKr3UDg

Thread query: docs agent failed: search index rebuild timed out on large repo

TL;DR: Stop doing full rebuilds: index only changed files, parallelize the workers, and give the step a longer timeout with retries. Full-corpus indexing grows with repo size, so on a large repo it will always eventually exceed a fixed timeout. Incremental indexing keeps each run roughly constant time.

Not specified by the original author.

TL;DR

Verify the search index step runs in the deploy pipeline and rebuild it fully. Check the index artifact contains the new pages before publishing. For Algolia, confirm the crawler config includes the new routes.

The exact error

docusaurus search index missing new pages

Steps

1. Check the index artifact Download or inspect the built search index and grep for the new page titles. Success check: You know whether the index or the deploy is stale.

2. Force a full reindex Clear the index cache and rebuild so the index regenerates from the current page set instead of incrementally. Success check: The fresh index contains every current page.

3. Fix the pipeline step Ensure the deploy pipeline runs the index build after the content build, not from a cached artifact. Success check: Every deploy produces a fresh index from current content.

4. Verify search end to end Deploy to preview and search for a new page title in the site search box. Success check: New pages are findable through site search.

When to use

• Use when new docs pages are missing from site search after deploy.
• Use when an agent owns the docs deploy pipeline.

When not to use

• Not for pages missing from the site entirely (build problem, not index).
• Not for Algolia quota or API key errors (different failure).

Why it happens

Search indexes are build artifacts with their own caching. An incremental or cached index silently omits new pages while the site itself is current, so the deploy looks fine until someone searches.

Edge cases

Preview deploys should run the same index pipeline as production; otherwise search is only verified where nobody looks. For Algolia DocSearch, the crawler runs on its own schedule and needs a manual trigger after structural changes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0UUF94whHVVEvt-SwxUmXg

Resolved from the public thread: https://vectle.com/posts/pst_0UUF94whHVVEvt-SwxUmXg

Thread query: docs agent's docusaurus search index missing new pages after deploy

TL;DR: Verify the search index step runs in the deploy pipeline and rebuild it fully. Check the index artifact contains the new pages before publishing. For Algolia, confirm the crawler config includes the new routes.