docs agent hallucinated parameters, published broken api docs

TL;DR

Diff every documented parameter against the real signature pulled from the code with inspect or your language's equivalent, and block publishing on zero mismatches. The agent invented plausible-sounding parameters because nothing checked its output against reality. Signature-diffing turns hallucinations into build failures instead of published lies.

docs agent hallucinated parameters, published broken api docs

Steps

1. Pull the ground truth for one suspect function: python -c "import inspect; import mypackage.mymodule; print(inspect.signature(mypackage.mymodule.my_function))". Expected: the real parameter list, straight from the code.
2. Compare against the generated docs: every documented parameter must appear in the signature with a compatible name. Expected: a concrete list of invented parameters (in docs, not in code) and any missing real ones.
3. Remove the invented parameters and add any missing real ones, then rebuild the docs. Expected: the regenerated page matches the signature exactly.
4. Add the gate to the agent loop: after generating API reference, the agent extracts documented parameters per function and diffs them against live signatures; any mismatch fails the run. Expected: a test run with a deliberately invented parameter exits nonzero naming the parameter.
5. Republish only after the gate passes clean. Expected: the published reference matches the code.

Use this when

• generated docs list parameters the code does not accept
• users report "unexpected keyword argument" after following your docs
• you need a verification step for agent-written API reference

Not for this skill when

• parameters are real but described wrongly (content quality issue)
• signatures are correct and docs are stale for other reasons (sync problem)
• a human wrote the docs (review process problem, not a hallucination gate)

Variant phrasings

• agent invented function parameters in docs
• api docs show parameters that do not exist
• hallucinated arguments in generated reference

Why it happens

Language models complete patterns: shown a function with three parameters and a docstring convention, the model happily writes five, drawing on similar functions it has seen. Without a mechanical check against the actual signature, the invented parameters read convincingly and ship.

Edge cases

• Variadic args legitimately accept anything: document them as variadic instead of flagging every call-site keyword as invented.
• Decorators can rewrite signatures: inspect the wrapped function so the ground truth is what callers actually see.
• Overloaded functions have multiple valid signatures: check against the union, not just the first.
• C-extension functions may have signatures inspect cannot read: fall back to stub files or headers as ground truth.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstrt2rYoeUKwsuw3yJ68ATw