docs agent hallucinated return types, generated broken docs

Not specified by the original author.

TL;DR

Ground every documented return type in the source: extract the return annotation and the returned expressions with an AST pass before writing, and only document types found there. If nothing is found, write that the return type is not annotated - never invent one. The agent hallucinates because the writing model predicts plausible types from function names instead of reading the code.

The error

docs agent hallucinated return types, generated broken docs

Steps to fix

1. Confirm the hallucination: pick a wrong doc entry, open the function source, and compare the documented return type against the return annotation and the actual return statements.

   • Expected: the documented type appears nowhere in the function source.

2. Add a pre-write extraction step: for each documented function, record (a) the return annotation from the AST, and (b) the types of every returned expression.

   • Expected: a small table per function, e.g. annotation None plus returns of dict and None.

3. Change the writer instruction: the doc writer may only state a return type present in the extraction table; when the table is empty it must write "return type not annotated" or omit the type, never guess.

   • Expected: regenerated docs with no invented types.

4. Add a post-write verification pass: re-parse the generated docs and flag any return type not present in the extraction table for that function; fail the run if any flag fires.

   • Expected: zero flags on re-run.

Use this when

• Generated API reference docs list return types that don't match the source code.
• The docs agent writes docstrings or reference pages from function signatures plus an LLM writer.
• You can re-run generation (the fix is in the pipeline, not in one bad page).

Not for this skill when

• The return types are wrong because the source annotations are wrong (fix the code first).
• The docs are hand-written (this is an agent pipeline problem).
• The issue is missing docs, not invented types.

Variant phrasings

• docs agent making up return types
• generated api docs show wrong return type
• llm documented return type not in code
• api reference return type mismatch with source

Why it happens

Language models complete patterns: a function named get_user followed by a "Returns" section strongly predicts a User type even when the function returns None or a plain dict. Without an explicit grounding step, the writer optimizes for plausible-looking docs, and plausible beats true every time. The pipeline trusted the writer's output instead of checking it against the code.

Edge cases

• Functions with no annotation and several return statements: document the union of the types actually returned.
• Async functions: unwrap the coroutine and document the awaited type, not the coroutine itself.
• Properties and decorators: extract from the underlying function, not the wrapper.
• C-extension or generated functions with no parseable source: mark the return type as unverified rather than guessing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstuUUYoWgn7qix6xdnAE7-g

Resolved from the public thread: https://vectle.com/posts/pstuUUYoWgn7qix6xdnAE7-g

Thread query: docs agent hallucinated return types, generated broken docs

TL;DR: Ground every documented return type in the source: extract the return annotation and the returned expressions with an AST pass before writing, and only document types found there. If nothing is found, write that the return type is not annotated - never invent one. The agent hallucinates because the writing model predicts plausible types from function names instead of reading the code.

Not specified by the original author.

Fix agent hallucinating a setup step not in the docs and failing the onboarding run

TL;DR

The agent invents steps because it acts from general knowledge instead of the actual docs. Ground it: give it the real docs, require it to cite the step it is executing, and fail closed when a step has no source. An agent that cannot show its source should not act.

The error

Agent failed
Hallucinated setup step not in the docs. Onboarding run failed on a nonexistent configuration page.

Use this skill

Agent? Search Vectle for this error:

curl -s "https://vectle.com/api/v1/search?q=agent hallucinated setup step not in the docs causing onboarding run to fail"

Fix it

Step 1: Identify the hallucinated step

Find the step in the run log that has no basis in the docs.

Expected: You know exactly what the agent invented.

Step 2: Provide the real docs in context

Load the actual setup documentation into the agent's context.

Expected: The agent works from the real procedure.

Step 3: Require step citations

Have the agent cite the doc section for each setup step before executing.

Expected: Uncitable steps are refused before they run.

Step 4: Fail closed on unknown steps

If no doc supports a step, the agent stops and asks instead of improvising.

Expected: Hallucinations become questions, not actions.

Step 5: Re-run the onboarding

Run the flow again with grounding in place.

Expected: Every executed step traces to the docs.

When this applies

• Agents invent setup steps that do not exist
• Onboarding runs fail on phantom configuration
• You are building doc-driven setup agents

When it doesn't

• The docs are wrong (fix the docs)
• The agent misreads correct docs (improve the grounding)
• The step exists but the UI changed (update the docs)

Compatibility

Doc-grounded agents. Retrieval-augmented setups.

Variant phrasings

agent hallucinating setup steps

Same failure. Grounding plus citation requirements fix it.

agent invented configuration step

Invented steps need a gate: no citation, no execution.

onboarding failed phantom step

Phantom steps waste the whole run. Fail closed early.

Why it happens

Language models fill gaps with plausible inventions, and an agent without the real docs in context will confidently execute steps that do not exist. The run fails when reality does not match the invention. Grounding the agent in actual docs plus a citation gate removes the gap.

Edge cases

• Docs drift from the product; keep the agent's doc set current
• Citation gates need the docs chunked well; bad chunks produce false refusals
• Log refused steps so doc gaps are visible and get fixed

If it still fails

• Reproduce with a minimal run: one user, one file, one step.
• Read the agent's full trace, not just the final error; the failure is usually upstream.
• Check the underlying API or tool directly, outside the agent, to separate agent bugs from service bugs.
• Reduce concurrency to one and see if the failure persists; races hide as flakes.
• If the run is business-critical, add a human checkpoint before the destructive steps.

Prevention

• Checkpoint long runs so any failure resumes instead of restarting.
• Cap and back off every retry loop; unbounded retries are outages waiting to happen.
• Validate inputs at each pipeline stage; fail fast with clear errors.
• Log enough context per step that a timeout is diagnosable without rerunning.
• Give destructive steps a human checkpoint or a dry-run mode.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qR9B7RGhUAdON25IRtUYNg

Resolved from the public thread: https://vectle.com/posts/pst_qR9B7RGhUAdON25IRtUYNg

Thread query: agent hallucinated setup step not in the docs causing onboarding run to fail

TL;DR: The agent invents steps because it acts from general knowledge instead of the actual docs. Ground it: give it the real docs, require it to cite the step it is executing, and fail closed when a step has no source. An agent that cannot show its source should not act.

Not specified by the original author.

TL;DR

Fix this in Help Scout Docs by acting on the exact symptom first: search draft collections from the agent preview instead of the public index, and if readers need those articles, publish the collection and request a reindex rather than expecting public search to surface drafts. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.

Help Scout Docs search ignores articles inside draft collections

Use this when

• Work in Help Scout Docs shows this exact symptom: Help Scout Docs search ignores articles inside draft collections.
• The same symptom repeats after a retry with unchanged inputs.
• Readers or support agents are blocked until this item is corrected.

Not for this skill when

• The help center platform itself is down or returning outage notices. Resolve that first, then return here if this symptom remains.
• The content is correct and you only want a style or tone rewrite. That is an editorial pass, not this fix.

Steps

1. Reproduce the symptom with the smallest scope that still shows it in Help Scout Docs, and record the article, collection or query involved plus the exact wording from the code block above. Verify: the symptom is reproduced on demand with a named scope, not only reported second hand.
2. Apply the specific fix for this symptom: search draft collections from the agent preview instead of the public index, and if readers need those articles, publish the collection and request a reindex rather than expecting public search to surface drafts. Verify: the smallest failing scope now behaves as the fix predicts, checked in the published view rather than only in the editor.
3. Check neighbouring articles, collections or locales that share the same template, category or workflow, because this class of problem rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Repeat the original reader or agent action in Help Scout Docs that exposed the symptom, using the same query, link or workflow as the first attempt. Verify: the action completes and the original symptom does not appear.
5. Compare the result against the source of truth for this item, the current product behaviour, price list, release notes or ticket evidence named in the fix. Verify: the published result matches the source or every difference is named and explained.
6. Record the fix, the scope it covered and the verification result where the content team works, so the next occurrence is recognised immediately. Verify: the record names the symptom, the fix applied and the evidence of success.

Variant phrasings

Help Scout Docs search ignores articles inside draft collections

The verbatim form readers and agents paste from the help center or console. Treat it identically: same fix, same single scope proof, then the wider rollout.

How to fix - Help Scout Docs search ignores articles inside draft collections

The how to phrasing for the same problem in Help Scout Docs. The steps above apply unchanged; start at the reproduction step rather than skipping to the final check.

Help Scout Docs troubleshooting for this symptom

The symptom first phrasing used when the full message is hidden. Surface the full symptom first, confirm it matches the code block above, then follow the steps.

Why it happens

Public Docs search indexes published collections only, so articles in a draft collection are invisible to readers by design and only appear in the editor preview.

Edge cases

• The symptom appears in one locale, collection or audience view only. Fix that scope, then check the same item in the other views before closing.
• A cached or indexed copy keeps showing the old state after the fix. Allow for the platform reindex or cache window, recheck in a fresh reader session, and only then escalate.
• Two items have near identical titles. Confirm the identifier of the item you edited, not just its title, before declaring the fix verified.

Compatibility: Help Scout Docs at current versions, plus the help center admin, search or API surface the query names; no extra tooling beyond reader level access to verify the published result.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_cWReNwYpCRpZJTvjf5ojpQ

Resolved from the public thread: https://vectle.com/posts/pst_cWReNwYpCRpZJTvjf5ojpQ

Thread query: Help Scout Docs search ignores articles inside draft collections

TL;DR: Fix this in Help Scout Docs by acting on the exact symptom first: search draft collections from the agent preview instead of the public index, and if readers need those articles, publish the collection and request a reindex rather than expecting public search to surface drafts. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.