docs agent failed: search index stale after docs deploy to s3
Fixes stale search indexes after docs deploys to S3. Use when site search returns old content after a deploy. Not for build failures.
TL;DR
Rebuild the search index as part of the deploy and invalidate the CDN cache. A stale index means the deploy updated pages but not the index the search reads.
Error
(site search returns last month's content after a successful deploy; no error text)Steps
- Confirm the staleness: search for a term from a just-published page. If the page is live but unsearchable, the index is stale, not the deploy. Expected: staleness confirmed.
- Check the deploy pipeline: the search index build (local search plugin output, Algolia crawl, or custom index JSON) must run on every deploy, not on a separate schedule. Expected: the index step located in the pipeline.
- Move index generation into the deploy job so it always runs with fresh content, and invalidate the CDN distribution after upload so edge caches serve the new index. Expected: index and content deploy atomically.
- Re-deploy and search again. Expected: new content is searchable immediately.
When to use
- Search results lag the published site
- The index is built by a separate cron or manual step
- Docs deploy to S3 behind a CDN
When not to use
- Search is broken entirely (an index configuration problem)
- The deploy itself failed (fix the deploy)
- You use Algolia DocSearch (their crawler schedule controls freshness)
Tool compatibility
Docusaurus/MkDocs/Sphinx search plugins with S3 plus CloudFront or similar CDN. Same pattern for any static host.
Variant phrasings
index fresh but CDN serves old index file
The invalidation step is missing or scoped to the wrong paths. Invalidate the index file path explicitly.
search stale only for one locale or version
The index build covers only part of the site. Extend it to all deployed variants.
Why it happens
Content deploy and index build are separate steps owned by different jobs, so they drift. Agents that "deploy the docs" upload HTML but never rebuild the artifact the search box reads.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_uFDVQhQwmE4espo486y0bA