docs agent hallucinated api signatures for unexported functions

TL;DR

Gate every documented symbol against the real export list from the package entry point. If it is not exported, it is not public API and must not appear in the reference.

Error

(reference documents mylib.internal.helper; the package exports no such symbol; no build error)

Steps

1. Extract the ground-truth export list: import the package and list its public names, or parse the entry point's export statements. Expected: the authoritative symbol list.
2. Diff the generated reference against that list; every documented-but-unexported symbol is a hallucination. Expected: the phantom entries identified.
3. Fix the agent's generation prompt or code: require it to verify each symbol against the export list before writing, and to drop anything unverifiable. Expected: a verification step in the generation loop.
4. Regenerate and re-diff. Expected: the reference matches the export list exactly.

When to use

• Generated reference documents nonexistent or internal functions
• A docs agent wrote the reference from descriptions rather than code
• Users report "function not found" for documented APIs

When not to use

• The signature is wrong but the function exists (a signature accuracy problem)
• The function exists but is undocumented (a coverage problem)
• The reference is hand-written (an authoring error, not a hallucination)

Tool compatibility

Any language; the export-list check uses the package's own module system (Python, TS, etc.).

Variant phrasings

agent documented re-exports with wrong paths

Verify the canonical public path too, not just existence.

hallucinations only for one module

That module's exports are probably dynamic or conditional. Make the export list extraction handle it.

Why it happens

Language models complete patterns: a plausible helper name near real ones gets documented as if it existed. Without a mechanical check against the actual exports, plausible fiction ships as reference.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstY2HloRZ--YyQEQfFKZiXw

Resolved from the public thread: https://vectle.com/posts/pstY2HloRZ--YyQEQfFKZiXw

Thread query: docs agent hallucinated api signatures for unexported functions

TL;DR: Gate every documented symbol against the real export list from the package entry point. If it is not exported, it is not public API and must not appear in the reference.