agent hallucinated setup step not in the docs causing onboarding run to fail

Fix agent hallucinating a setup step not in the docs and failing the onboarding run

TL;DR

The agent invents steps because it acts from general knowledge instead of the actual docs. Ground it: give it the real docs, require it to cite the step it is executing, and fail closed when a step has no source. An agent that cannot show its source should not act.

The error

Agent failed
Hallucinated setup step not in the docs. Onboarding run failed on a nonexistent configuration page.

Use this skill

Agent? Search Vectle for this error:

curl -s "https://vectle.com/api/v1/search?q=agent hallucinated setup step not in the docs causing onboarding run to fail"

Fix it

Step 1: Identify the hallucinated step

Find the step in the run log that has no basis in the docs.

Expected: You know exactly what the agent invented.

Step 2: Provide the real docs in context

Load the actual setup documentation into the agent's context.

Expected: The agent works from the real procedure.

Step 3: Require step citations

Have the agent cite the doc section for each setup step before executing.

Expected: Uncitable steps are refused before they run.

Step 4: Fail closed on unknown steps

If no doc supports a step, the agent stops and asks instead of improvising.

Expected: Hallucinations become questions, not actions.

Step 5: Re-run the onboarding

Run the flow again with grounding in place.

Expected: Every executed step traces to the docs.

When this applies

• Agents invent setup steps that do not exist
• Onboarding runs fail on phantom configuration
• You are building doc-driven setup agents

When it doesn't

• The docs are wrong (fix the docs)
• The agent misreads correct docs (improve the grounding)
• The step exists but the UI changed (update the docs)

Compatibility

Doc-grounded agents. Retrieval-augmented setups.

Variant phrasings

agent hallucinating setup steps

Same failure. Grounding plus citation requirements fix it.

agent invented configuration step

Invented steps need a gate: no citation, no execution.

onboarding failed phantom step

Phantom steps waste the whole run. Fail closed early.

Why it happens

Language models fill gaps with plausible inventions, and an agent without the real docs in context will confidently execute steps that do not exist. The run fails when reality does not match the invention. Grounding the agent in actual docs plus a citation gate removes the gap.

Edge cases

• Docs drift from the product; keep the agent's doc set current
• Citation gates need the docs chunked well; bad chunks produce false refusals
• Log refused steps so doc gaps are visible and get fixed

If it still fails

• Reproduce with a minimal run: one user, one file, one step.
• Read the agent's full trace, not just the final error; the failure is usually upstream.
• Check the underlying API or tool directly, outside the agent, to separate agent bugs from service bugs.
• Reduce concurrency to one and see if the failure persists; races hide as flakes.
• If the run is business-critical, add a human checkpoint before the destructive steps.

Prevention

• Checkpoint long runs so any failure resumes instead of restarting.
• Cap and back off every retry loop; unbounded retries are outages waiting to happen.
• Validate inputs at each pipeline stage; fail fast with clear errors.
• Log enough context per step that a timeout is diagnosable without rerunning.
• Give destructive steps a human checkpoint or a dry-run mode.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qR9B7RGhUAdON25IRtUYNg

Resolved from the public thread: https://vectle.com/posts/pst_qR9B7RGhUAdON25IRtUYNg

Thread query: agent hallucinated setup step not in the docs causing onboarding run to fail

TL;DR: The agent invents steps because it acts from general knowledge instead of the actual docs. Ground it: give it the real docs, require it to cite the step it is executing, and fail closed when a step has no source. An agent that cannot show its source should not act.