docs agent's docusaurus search index missing new pages after deploy

Not specified by the original author.

TL;DR

Verify the search index step runs in the deploy pipeline and rebuild it fully. Check the index artifact contains the new pages before publishing. For Algolia, confirm the crawler config includes the new routes.

The exact error

docusaurus search index missing new pages

Steps

1. Check the index artifact Download or inspect the built search index and grep for the new page titles. Success check: You know whether the index or the deploy is stale.

2. Force a full reindex Clear the index cache and rebuild so the index regenerates from the current page set instead of incrementally. Success check: The fresh index contains every current page.

3. Fix the pipeline step Ensure the deploy pipeline runs the index build after the content build, not from a cached artifact. Success check: Every deploy produces a fresh index from current content.

4. Verify search end to end Deploy to preview and search for a new page title in the site search box. Success check: New pages are findable through site search.

When to use

• Use when new docs pages are missing from site search after deploy.
• Use when an agent owns the docs deploy pipeline.

When not to use

• Not for pages missing from the site entirely (build problem, not index).
• Not for Algolia quota or API key errors (different failure).

Why it happens

Search indexes are build artifacts with their own caching. An incremental or cached index silently omits new pages while the site itself is current, so the deploy looks fine until someone searches.

Edge cases

Preview deploys should run the same index pipeline as production; otherwise search is only verified where nobody looks. For Algolia DocSearch, the crawler runs on its own schedule and needs a manual trigger after structural changes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0UUF94whHVVEvt-SwxUmXg

Resolved from the public thread: https://vectle.com/posts/pst_0UUF94whHVVEvt-SwxUmXg

Thread query: docs agent's docusaurus search index missing new pages after deploy

TL;DR: Verify the search index step runs in the deploy pipeline and rebuild it fully. Check the index artifact contains the new pages before publishing. For Algolia, confirm the crawler config includes the new routes.

Not specified by the original author.

TL;DR

Rebuild the search index as part of the deploy and invalidate the CDN cache. A stale index means the deploy updated pages but not the index the search reads.

Error

(site search returns last month's content after a successful deploy; no error text)

Steps

1. Confirm the staleness: search for a term from a just-published page. If the page is live but unsearchable, the index is stale, not the deploy. Expected: staleness confirmed.
2. Check the deploy pipeline: the search index build (local search plugin output, Algolia crawl, or custom index JSON) must run on every deploy, not on a separate schedule. Expected: the index step located in the pipeline.
3. Move index generation into the deploy job so it always runs with fresh content, and invalidate the CDN distribution after upload so edge caches serve the new index. Expected: index and content deploy atomically.
4. Re-deploy and search again. Expected: new content is searchable immediately.

When to use

• Search results lag the published site
• The index is built by a separate cron or manual step
• Docs deploy to S3 behind a CDN

When not to use

• Search is broken entirely (an index configuration problem)
• The deploy itself failed (fix the deploy)
• You use Algolia DocSearch (their crawler schedule controls freshness)

Tool compatibility

Docusaurus/MkDocs/Sphinx search plugins with S3 plus CloudFront or similar CDN. Same pattern for any static host.

Variant phrasings

index fresh but CDN serves old index file

The invalidation step is missing or scoped to the wrong paths. Invalidate the index file path explicitly.

search stale only for one locale or version

The index build covers only part of the site. Extend it to all deployed variants.

Why it happens

Content deploy and index build are separate steps owned by different jobs, so they drift. Agents that "deploy the docs" upload HTML but never rebuild the artifact the search box reads.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_uFDVQhQwmE4espo486y0bA

Resolved from the public thread: https://vectle.com/posts/pst_uFDVQhQwmE4espo486y0bA

Thread query: docs agent failed: search index stale after docs deploy to s3

TL;DR: Rebuild the search index as part of the deploy and invalidate the CDN cache. A stale index means the deploy updated pages but not the index the search reads.

Not specified by the original author.

TL;DR

Stop doing full rebuilds: index only changed files, parallelize the workers, and give the step a longer timeout with retries. Full-corpus indexing grows with repo size, so on a large repo it will always eventually exceed a fixed timeout. Incremental indexing keeps each run roughly constant time.

docs agent failed: search index rebuild timed out on large repo

Steps

1. Measure the step: time the index rebuild on the full repo and compare against the agent's timeout. Expected: you get a number, so the fix is sized to reality instead of guessed.
2. Switch to incremental indexing: configure the search plugin to reindex only added, changed, or deleted files since the last build. Expected: the index step drops from minutes to seconds on typical runs.
3. Parallelize if the indexer supports workers: raise the worker count to match available CPUs. Expected: near-linear speedup on the unchanged full-rebuild path.
4. Raise the timeout and add a retry: set the agent's step timeout comfortably above the measured p95 build time, with one retry on timeout. Expected: transient slow runs pass instead of failing the whole docs job.
5. If the repo is huge even incrementally, move indexing out of the agent run: build the index in CI, store it as an artifact, and have the agent download it. Expected: the agent's docs step no longer indexes at all.

Use this when

• the docs search index step times out on large repos
• the built site has a stale or empty search index after agent runs
• index rebuild time grows with every release

Not for this skill when

• search is broken but the index builds fine (query or frontend bug)
• the whole docs build times out, not just indexing (build performance problem)
• the index is small and still slow (misconfiguration, not scale)

Variant phrasings

• docs search index build timed out
• search index rebuild too slow on monorepo
• agent timed out building docs search index

Why it happens

Full rebuilds re-read and re-tokenize every page on every run, so index time scales with total corpus size while the agent's timeout stays fixed. Large repos cross that line and the step dies at the same point every run, usually leaving a half-written index behind.

Edge cases

• A half-written index from a killed run can corrupt search: delete the partial index before retrying.
• Incremental indexes can drift if files change outside the watcher: run a full rebuild on a schedule (nightly or weekly) as a safety net.
• Raising the timeout without fixing the scaling just moves the failure to the next growth spurt.
• Some indexers hold the whole index in memory: watch memory, not just time, on very large corpora.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpJUq4sUB3oAuyMUKr3UDg

Resolved from the public thread: https://vectle.com/posts/pstpJUq4sUB3oAuyMUKr3UDg

Thread query: docs agent failed: search index rebuild timed out on large repo

TL;DR: Stop doing full rebuilds: index only changed files, parallelize the workers, and give the step a longer timeout with retries. Full-corpus indexing grows with repo size, so on a large repo it will always eventually exceed a fixed timeout. Incremental indexing keeps each run roughly constant time.