docs agent failed: file watcher missed new modules during generation

TL;DR

Do a full reconciliation pass after every watch-driven generation: list all real modules and confirm each one has a generated page. Watchers miss files because of ignore rules, symlinked directories, or OS event limits, so never trust the event stream alone. The reconcile pass turns a silent skip into a loud failure.

docs agent failed: file watcher missed new modules during generation

Steps

1. Prove the miss: list real modules with git ls-files '*.py' (or the equivalent glob for your language) and compare against the generated reference pages. Expected: a one-to-one match. Every module without a page is a miss.
2. Check the watch root: confirm the watcher is rooted at the repo root or the package source dir, not at a subdirectory that excludes new packages. New top-level packages are the classic miss.
3. Check ignore rules: look for ignore patterns that swallow new directories (broad excludes, stale ignore lists, symlinked package dirs that the watcher does not follow). Expected: no ignore rule matches your source dirs.
4. Turn on polling as a fallback if you are on a network filesystem or container mount where OS events are unreliable. Expected: file creation events now fire for new modules.
5. Add the reconcile gate: after generation, the agent enumerates every module and fails the run if any module lacks a page. Expected: a run with a missed module exits nonzero with the module name in the message.

Use this when

• new modules have no generated reference pages after a watch-driven run
• the watcher log shows no errors but pages are missing
• generation works for edits but not for newly added files
• docs are stale only for recently added code

Not for this skill when

• pages exist but content is wrong (generation bug, not a watcher bug)
• the build itself fails (builder error)
• modules are excluded on purpose by config

Variant phrasings

• watcher did not pick up new files during docs generation
• generated docs missing new modules
• file watcher ignoring new source files
• docs agent skipped newly added module

Why it happens

File watchers deliver best-effort event streams: OS limits on watch handles, ignored symlinked dirs, overly broad ignore globs, and event coalescing on fast checkouts all drop creation events silently. A generation pipeline that only reacts to events therefore has no idea what it never saw.

Edge cases

• Watch handle exhaustion on Linux surfaces as ENOSPC errors in the watcher log: raise the max watch handle limit or narrow the watch root.
• Polling fixes reliability but costs CPU on huge trees: scope polling to source dirs only.
• Generated files triggering the watcher can cause infinite regen loops: ignore the output dir.
• Case-only renames on case-insensitive filesystems may not fire events: the reconcile pass catches these too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GgUuzR-mTP4YTDTI4Gsa9Q