docs agent hallucinated return types, generated broken docs

TL;DR

Ground every documented return type in the source: extract the return annotation and the returned expressions with an AST pass before writing, and only document types found there. If nothing is found, write that the return type is not annotated - never invent one. The agent hallucinates because the writing model predicts plausible types from function names instead of reading the code.

The error

docs agent hallucinated return types, generated broken docs

Steps to fix

1. Confirm the hallucination: pick a wrong doc entry, open the function source, and compare the documented return type against the return annotation and the actual return statements.

   • Expected: the documented type appears nowhere in the function source.

2. Add a pre-write extraction step: for each documented function, record (a) the return annotation from the AST, and (b) the types of every returned expression.

   • Expected: a small table per function, e.g. annotation None plus returns of dict and None.

3. Change the writer instruction: the doc writer may only state a return type present in the extraction table; when the table is empty it must write "return type not annotated" or omit the type, never guess.

   • Expected: regenerated docs with no invented types.

4. Add a post-write verification pass: re-parse the generated docs and flag any return type not present in the extraction table for that function; fail the run if any flag fires.

   • Expected: zero flags on re-run.

Use this when

• Generated API reference docs list return types that don't match the source code.
• The docs agent writes docstrings or reference pages from function signatures plus an LLM writer.
• You can re-run generation (the fix is in the pipeline, not in one bad page).

Not for this skill when

• The return types are wrong because the source annotations are wrong (fix the code first).
• The docs are hand-written (this is an agent pipeline problem).
• The issue is missing docs, not invented types.

Variant phrasings

• docs agent making up return types
• generated api docs show wrong return type
• llm documented return type not in code
• api reference return type mismatch with source

Why it happens

Language models complete patterns: a function named get_user followed by a "Returns" section strongly predicts a User type even when the function returns None or a plain dict. Without an explicit grounding step, the writer optimizes for plausible-looking docs, and plausible beats true every time. The pipeline trusted the writer's output instead of checking it against the code.

Edge cases

• Functions with no annotation and several return statements: document the union of the types actually returned.
• Async functions: unwrap the coroutine and document the awaited type, not the coroutine itself.
• Properties and decorators: extract from the underlying function, not the wrapper.
• C-extension or generated functions with no parseable source: mark the return type as unverified rather than guessing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstuUUYoWgn7qix6xdnAE7-g