docs agent hallucinated api signatures for unexported functions
Fixes docs agents that document API signatures for functions that are not actually exported. Use when generated reference lists functions the package does not export. Not for wrong signatures of real exports.
TL;DR
Gate every documented symbol against the real export list from the package entry point. If it is not exported, it is not public API and must not appear in the reference.
Error
(reference documents mylib.internal.helper; the package exports no such symbol; no build error)Steps
- Extract the ground-truth export list: import the package and list its public names, or parse the entry point's export statements. Expected: the authoritative symbol list.
- Diff the generated reference against that list; every documented-but-unexported symbol is a hallucination. Expected: the phantom entries identified.
- Fix the agent's generation prompt or code: require it to verify each symbol against the export list before writing, and to drop anything unverifiable. Expected: a verification step in the generation loop.
- Regenerate and re-diff. Expected: the reference matches the export list exactly.
When to use
- Generated reference documents nonexistent or internal functions
- A docs agent wrote the reference from descriptions rather than code
- Users report "function not found" for documented APIs
When not to use
- The signature is wrong but the function exists (a signature accuracy problem)
- The function exists but is undocumented (a coverage problem)
- The reference is hand-written (an authoring error, not a hallucination)
Tool compatibility
Any language; the export-list check uses the package's own module system (Python, TS, etc.).
Variant phrasings
agent documented re-exports with wrong paths
Verify the canonical public path too, not just existence.
hallucinations only for one module
That module's exports are probably dynamic or conditional. Make the export list extraction handle it.
Why it happens
Language models complete patterns: a plausible helper name near real ones gets documented as if it existed. Without a mechanical check against the actual exports, plausible fiction ships as reference.
Provenance
Resolved from the public thread: https://vectle.com/posts/pstY2HloRZ--YyQEQfFKZiXw