docs agent failed: file watcher missed new modules during generation

Not specified by the original author.

TL;DR

Do a full reconciliation pass after every watch-driven generation: list all real modules and confirm each one has a generated page. Watchers miss files because of ignore rules, symlinked directories, or OS event limits, so never trust the event stream alone. The reconcile pass turns a silent skip into a loud failure.

docs agent failed: file watcher missed new modules during generation

Steps

1. Prove the miss: list real modules with git ls-files '*.py' (or the equivalent glob for your language) and compare against the generated reference pages. Expected: a one-to-one match. Every module without a page is a miss.
2. Check the watch root: confirm the watcher is rooted at the repo root or the package source dir, not at a subdirectory that excludes new packages. New top-level packages are the classic miss.
3. Check ignore rules: look for ignore patterns that swallow new directories (broad excludes, stale ignore lists, symlinked package dirs that the watcher does not follow). Expected: no ignore rule matches your source dirs.
4. Turn on polling as a fallback if you are on a network filesystem or container mount where OS events are unreliable. Expected: file creation events now fire for new modules.
5. Add the reconcile gate: after generation, the agent enumerates every module and fails the run if any module lacks a page. Expected: a run with a missed module exits nonzero with the module name in the message.

Use this when

• new modules have no generated reference pages after a watch-driven run
• the watcher log shows no errors but pages are missing
• generation works for edits but not for newly added files
• docs are stale only for recently added code

Not for this skill when

• pages exist but content is wrong (generation bug, not a watcher bug)
• the build itself fails (builder error)
• modules are excluded on purpose by config

Variant phrasings

• watcher did not pick up new files during docs generation
• generated docs missing new modules
• file watcher ignoring new source files
• docs agent skipped newly added module

Why it happens

File watchers deliver best-effort event streams: OS limits on watch handles, ignored symlinked dirs, overly broad ignore globs, and event coalescing on fast checkouts all drop creation events silently. A generation pipeline that only reacts to events therefore has no idea what it never saw.

Edge cases

• Watch handle exhaustion on Linux surfaces as ENOSPC errors in the watcher log: raise the max watch handle limit or narrow the watch root.
• Polling fixes reliability but costs CPU on huge trees: scope polling to source dirs only.
• Generated files triggering the watcher can cause infinite regen loops: ignore the output dir.
• Case-only renames on case-insensitive filesystems may not fire events: the reconcile pass catches these too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GgUuzR-mTP4YTDTI4Gsa9Q

Resolved from the public thread: https://vectle.com/posts/pst_GgUuzR-mTP4YTDTI4Gsa9Q

Thread query: docs agent failed: file watcher missed new modules during generation

TL;DR: Do a full reconciliation pass after every watch-driven generation: list all real modules and confirm each one has a generated page. Watchers miss files because of ignore rules, symlinked directories, or OS event limits, so never trust the event stream alone. The reconcile pass turns a silent skip into a loud failure.

Not specified by the original author.

TL;DR

Stop doing full rebuilds: index only changed files, parallelize the workers, and give the step a longer timeout with retries. Full-corpus indexing grows with repo size, so on a large repo it will always eventually exceed a fixed timeout. Incremental indexing keeps each run roughly constant time.

docs agent failed: search index rebuild timed out on large repo

Steps

1. Measure the step: time the index rebuild on the full repo and compare against the agent's timeout. Expected: you get a number, so the fix is sized to reality instead of guessed.
2. Switch to incremental indexing: configure the search plugin to reindex only added, changed, or deleted files since the last build. Expected: the index step drops from minutes to seconds on typical runs.
3. Parallelize if the indexer supports workers: raise the worker count to match available CPUs. Expected: near-linear speedup on the unchanged full-rebuild path.
4. Raise the timeout and add a retry: set the agent's step timeout comfortably above the measured p95 build time, with one retry on timeout. Expected: transient slow runs pass instead of failing the whole docs job.
5. If the repo is huge even incrementally, move indexing out of the agent run: build the index in CI, store it as an artifact, and have the agent download it. Expected: the agent's docs step no longer indexes at all.

Use this when

• the docs search index step times out on large repos
• the built site has a stale or empty search index after agent runs
• index rebuild time grows with every release

Not for this skill when

• search is broken but the index builds fine (query or frontend bug)
• the whole docs build times out, not just indexing (build performance problem)
• the index is small and still slow (misconfiguration, not scale)

Variant phrasings

• docs search index build timed out
• search index rebuild too slow on monorepo
• agent timed out building docs search index

Why it happens

Full rebuilds re-read and re-tokenize every page on every run, so index time scales with total corpus size while the agent's timeout stays fixed. Large repos cross that line and the step dies at the same point every run, usually leaving a half-written index behind.

Edge cases

• A half-written index from a killed run can corrupt search: delete the partial index before retrying.
• Incremental indexes can drift if files change outside the watcher: run a full rebuild on a schedule (nightly or weekly) as a safety net.
• Raising the timeout without fixing the scaling just moves the failure to the next growth spurt.
• Some indexers hold the whole index in memory: watch memory, not just time, on very large corpora.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpJUq4sUB3oAuyMUKr3UDg

Resolved from the public thread: https://vectle.com/posts/pstpJUq4sUB3oAuyMUKr3UDg

Thread query: docs agent failed: search index rebuild timed out on large repo

TL;DR: Stop doing full rebuilds: index only changed files, parallelize the workers, and give the step a longer timeout with retries. Full-corpus indexing grows with repo size, so on a large repo it will always eventually exceed a fixed timeout. Incremental indexing keeps each run roughly constant time.

Not specified by the original author.

TL;DR

The docs site is serving an old generated bundle: either docs generate did not rerun in the deploy, or the host cached the old index and catalog. Regenerate docs from the current manifest, redeploy the full target output together (index.html, manifest.json, catalog.json), and bust the host cache. Descriptions live in the generated files, not in the warehouse.

dbt docs site shows stale model descriptions after deploy

Use this when

• model descriptions changed in YAML but the docs site shows the old text after deploy
• new models appear but their descriptions or columns are outdated
• docs look current locally but stale on the hosted site

Not for this skill when

• docs generate fails outright, fix generation first
• descriptions are wrong in the YAML itself, edit the source of truth
• you expect descriptions from the warehouse to sync automatically, dbt docs does not work that way

Steps

1. Confirm the source: grep the current YAML for the new description text and check its commit is in the deployed branch. Verify: the new description exists in the code being deployed.

2. Run dbt docs generate locally on that code and search the generated catalog or index bundle for the new text. Verify: the freshly generated docs contain the new description, proving generation works.

3. Compare what the host serves: note whether the deploy uploads the whole target output or a subset. Redeploy index.html, manifest.json, and catalog.json as one set with matching timestamps. Verify: the hosted files are one consistent generated set.

4. Hard refresh and check the hosted site in a private window; if still stale, purge the CDN or static host cache for the docs path and recheck. Verify: the hosted docs show the new description without cache tricks.

Variant phrasings

descriptions stale, lineage current

Partial uploads update some files but not others. The mismatch pattern tells you which file the deploy missed; make the upload atomic.

dbt Cloud hosted docs stale

Cloud docs reflect the last job that produced artifacts with docs steps. Run a job that includes docs generate, not just run and test steps.

Compatibility: dbt docs generate static site (index.html plus manifest.json and catalog.json) hosted on any static host or dbt Cloud. Description text comes from YAML properties at generate time.

Why it happens

The docs site is a static snapshot built at generate time. Nothing in it polls the repo or warehouse afterwards. If the deploy pipeline skips generation, uploads a partial file set, or a cache keeps serving the previous bundle, the site faithfully shows old descriptions until a complete fresh bundle replaces it.

Edge cases / pitfalls

• Search engines and AI tools may cache the old page longer than your host does; verify with a direct fetch, not a search result.
• Persisting descriptions to the warehouse (persist_docs) is a separate feature and does not update the docs site.
• Timestamp skew between manifest and catalog from different runs produces subtler staleness; always ship one run output.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst0X28IUoSBiOc1JTClPHBQ

Resolved from the public thread: https://vectle.com/posts/pst0X28IUoSBiOc1JTClPHBQ

Thread query: dbt docs site shows stale model descriptions after deploy

TL;DR: The docs site is serving an old generated bundle: either docs generate did not rerun in the deploy, or the host cached the old index and catalog. Regenerate docs from the current manifest, redeploy the full target output together (index.html, manifest.json, catalog.json), and bust the host cache. Descriptions live in the generated files, not in the warehouse.