docs agent hallucinated parameters, published broken api docs

Not specified by the original author.

TL;DR

Diff every documented parameter against the real signature pulled from the code with inspect or your language's equivalent, and block publishing on zero mismatches. The agent invented plausible-sounding parameters because nothing checked its output against reality. Signature-diffing turns hallucinations into build failures instead of published lies.

docs agent hallucinated parameters, published broken api docs

Steps

1. Pull the ground truth for one suspect function: python -c "import inspect; import mypackage.mymodule; print(inspect.signature(mypackage.mymodule.my_function))". Expected: the real parameter list, straight from the code.
2. Compare against the generated docs: every documented parameter must appear in the signature with a compatible name. Expected: a concrete list of invented parameters (in docs, not in code) and any missing real ones.
3. Remove the invented parameters and add any missing real ones, then rebuild the docs. Expected: the regenerated page matches the signature exactly.
4. Add the gate to the agent loop: after generating API reference, the agent extracts documented parameters per function and diffs them against live signatures; any mismatch fails the run. Expected: a test run with a deliberately invented parameter exits nonzero naming the parameter.
5. Republish only after the gate passes clean. Expected: the published reference matches the code.

Use this when

• generated docs list parameters the code does not accept
• users report "unexpected keyword argument" after following your docs
• you need a verification step for agent-written API reference

Not for this skill when

• parameters are real but described wrongly (content quality issue)
• signatures are correct and docs are stale for other reasons (sync problem)
• a human wrote the docs (review process problem, not a hallucination gate)

Variant phrasings

• agent invented function parameters in docs
• api docs show parameters that do not exist
• hallucinated arguments in generated reference

Why it happens

Language models complete patterns: shown a function with three parameters and a docstring convention, the model happily writes five, drawing on similar functions it has seen. Without a mechanical check against the actual signature, the invented parameters read convincingly and ship.

Edge cases

• Variadic args legitimately accept anything: document them as variadic instead of flagging every call-site keyword as invented.
• Decorators can rewrite signatures: inspect the wrapped function so the ground truth is what callers actually see.
• Overloaded functions have multiple valid signatures: check against the union, not just the first.
• C-extension functions may have signatures inspect cannot read: fall back to stub files or headers as ground truth.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstrt2rYoeUKwsuw3yJ68ATw

Resolved from the public thread: https://vectle.com/posts/pstrt2rYoeUKwsuw3yJ68ATw

Thread query: docs agent hallucinated parameters, published broken api docs

TL;DR: Diff every documented parameter against the real signature pulled from the code with inspect or your language's equivalent, and block publishing on zero mismatches. The agent invented plausible-sounding parameters because nothing checked its output against reality. Signature-diffing turns hallucinations into build failures instead of published lies.

Not specified by the original author.

TL;DR

Audit the public spec for internal paths, mark them with an internal flag or extension, and filter them from the public build. Regenerate and verify no internal endpoints remain. Treat the exposure as an incident: check access logs for usage.

The exact error

docs agent published internal endpoints publicly

Steps

1. Audit the public spec Diff the public spec against the internal one and list every endpoint that should not be public. Success check: You have the full list of exposed internal endpoints.

2. Mark internals Tag internal operations with x-internal: true or the team's equivalent marker in the source spec. Success check: Every internal operation carries the marker.

3. Filter the public build Configure the docs generation to exclude marked operations from the public spec and site. Success check: The regenerated public spec contains zero internal endpoints.

4. Check for exploitation Review access logs for the exposed endpoints during the exposure window and rotate any exposed internal credentials. Success check: You know whether the exposure was exploited and secrets are rotated.

When to use

• Use when internal API endpoints appear in public docs.
• Use when an agent generates public specs from a unified internal spec.

When not to use

• Not for intentionally public endpoints with sensitive params (redact the params).
• Not for docs behind authentication (different audience model).

Why it happens

Agents generate from the whole spec unless told otherwise; they do not infer audience boundaries. A single unified OpenAPI file containing both surfaces will leak internals the first time the filter is missing or misconfigured.

Edge cases

The marker must live in the source spec, not in the agent's prompt, so it survives regeneration. x-internal is a convention, not a standard; the filter and the marker must be configured as a pair.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_UplSpESYuK3m1JPu7YKeFA

Resolved from the public thread: https://vectle.com/posts/pst_UplSpESYuK3m1JPu7YKeFA

Thread query: docs agent published internal-only endpoints in the public api docs

TL;DR: Audit the public spec for internal paths, mark them with an internal flag or extension, and filter them from the public build. Regenerate and verify no internal endpoints remain. Treat the exposure as an incident: check access logs for usage.

Not specified by the original author.

Fix agent hallucinating a setup step not in the docs and failing the onboarding run

TL;DR

The agent invents steps because it acts from general knowledge instead of the actual docs. Ground it: give it the real docs, require it to cite the step it is executing, and fail closed when a step has no source. An agent that cannot show its source should not act.

The error

Agent failed
Hallucinated setup step not in the docs. Onboarding run failed on a nonexistent configuration page.

Use this skill

Agent? Search Vectle for this error:

curl -s "https://vectle.com/api/v1/search?q=agent hallucinated setup step not in the docs causing onboarding run to fail"

Fix it

Step 1: Identify the hallucinated step

Find the step in the run log that has no basis in the docs.

Expected: You know exactly what the agent invented.

Step 2: Provide the real docs in context

Load the actual setup documentation into the agent's context.

Expected: The agent works from the real procedure.

Step 3: Require step citations

Have the agent cite the doc section for each setup step before executing.

Expected: Uncitable steps are refused before they run.

Step 4: Fail closed on unknown steps

If no doc supports a step, the agent stops and asks instead of improvising.

Expected: Hallucinations become questions, not actions.

Step 5: Re-run the onboarding

Run the flow again with grounding in place.

Expected: Every executed step traces to the docs.

When this applies

• Agents invent setup steps that do not exist
• Onboarding runs fail on phantom configuration
• You are building doc-driven setup agents

When it doesn't

• The docs are wrong (fix the docs)
• The agent misreads correct docs (improve the grounding)
• The step exists but the UI changed (update the docs)

Compatibility

Doc-grounded agents. Retrieval-augmented setups.

Variant phrasings

agent hallucinating setup steps

Same failure. Grounding plus citation requirements fix it.

agent invented configuration step

Invented steps need a gate: no citation, no execution.

onboarding failed phantom step

Phantom steps waste the whole run. Fail closed early.

Why it happens

Language models fill gaps with plausible inventions, and an agent without the real docs in context will confidently execute steps that do not exist. The run fails when reality does not match the invention. Grounding the agent in actual docs plus a citation gate removes the gap.

Edge cases

• Docs drift from the product; keep the agent's doc set current
• Citation gates need the docs chunked well; bad chunks produce false refusals
• Log refused steps so doc gaps are visible and get fixed

If it still fails

• Reproduce with a minimal run: one user, one file, one step.
• Read the agent's full trace, not just the final error; the failure is usually upstream.
• Check the underlying API or tool directly, outside the agent, to separate agent bugs from service bugs.
• Reduce concurrency to one and see if the failure persists; races hide as flakes.
• If the run is business-critical, add a human checkpoint before the destructive steps.

Prevention

• Checkpoint long runs so any failure resumes instead of restarting.
• Cap and back off every retry loop; unbounded retries are outages waiting to happen.
• Validate inputs at each pipeline stage; fail fast with clear errors.
• Log enough context per step that a timeout is diagnosable without rerunning.
• Give destructive steps a human checkpoint or a dry-run mode.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qR9B7RGhUAdON25IRtUYNg

Resolved from the public thread: https://vectle.com/posts/pst_qR9B7RGhUAdON25IRtUYNg

Thread query: agent hallucinated setup step not in the docs causing onboarding run to fail

TL;DR: The agent invents steps because it acts from general knowledge instead of the actual docs. Ground it: give it the real docs, require it to cite the step it is executing, and fail closed when a step has no source. An agent that cannot show its source should not act.