docs agent hallucinated api endpoints, generated broken links

Not specified by the original author.

TL;DR

Verify every endpoint the agent documents against the real route surface (your OpenAPI spec paths or the framework's route list) and run a link checker over the built docs before publishing. The agent linked endpoints it imagined because no step compared its output to the actual API. Route-checking turns invented endpoints into caught errors.

docs agent hallucinated api endpoints, generated broken links

Steps

1. Extract the endpoints from the generated docs: search the built pages or sources for path-like links (strings starting with /v1/, /api/, and similar in your project). Expected: a concrete list of documented endpoints.
2. Pull the real route list: from your OpenAPI spec's paths object, or your framework's route listing command. Expected: the authoritative set of endpoints that actually exist.
3. Diff the two lists: every documented endpoint must exist in the real routes. Expected: a concrete list of invented endpoints (in docs, not in routes).
4. Delete or correct the invented endpoints, fix the links, and rebuild. Expected: a link checker over the built site reports zero broken internal endpoint links.
5. Add the gate: the agent diffs documented endpoints against the route list before publishing, and any invented endpoint fails the run. Expected: a test run with a fake endpoint exits nonzero naming it.

Use this when

• generated docs link to endpoints that 404
• a link checker flags agent-generated endpoint URLs
• API reference was written by an agent, not a human

Not for this skill when

• endpoints are real but documented wrongly (content issue)
• links break because the API changed after publishing (drift; regenerate)
• humans wrote the docs (review problem)

Variant phrasings

• agent invented api endpoints in docs
• generated docs link to nonexistent endpoints
• broken endpoint links from docs agent

Why it happens

Endpoint paths are highly patterned, so a language model can extend the pattern into endpoints that look right but were never implemented. The docs read convincingly, the links resolve nowhere, and users discover it in production.

Edge cases

• Path parameters make naive string matching fail: compare route patterns, not literal URLs with sample ids.
• Versioned prefixes are a classic mismatch: check the version segment explicitly.
• Endpoints behind feature flags exist in code but not in prod: decide whether docs should list them at all.
• External links rot independently of hallucination: check those on a schedule too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstr8chTxbNWKTKW8kEtf3Xw

Resolved from the public thread: https://vectle.com/posts/pstr8chTxbNWKTKW8kEtf3Xw

Thread query: docs agent hallucinated api endpoints, generated broken links

TL;DR: Verify every endpoint the agent documents against the real route surface (your OpenAPI spec paths or the framework's route list) and run a link checker over the built docs before publishing. The agent linked endpoints it imagined because no step compared its output to the actual API. Route-checking turns invented endpoints into caught errors.

Not specified by the original author.

TL;DR

Audit the public spec for internal paths, mark them with an internal flag or extension, and filter them from the public build. Regenerate and verify no internal endpoints remain. Treat the exposure as an incident: check access logs for usage.

The exact error

docs agent published internal endpoints publicly

Steps

1. Audit the public spec Diff the public spec against the internal one and list every endpoint that should not be public. Success check: You have the full list of exposed internal endpoints.

2. Mark internals Tag internal operations with x-internal: true or the team's equivalent marker in the source spec. Success check: Every internal operation carries the marker.

3. Filter the public build Configure the docs generation to exclude marked operations from the public spec and site. Success check: The regenerated public spec contains zero internal endpoints.

4. Check for exploitation Review access logs for the exposed endpoints during the exposure window and rotate any exposed internal credentials. Success check: You know whether the exposure was exploited and secrets are rotated.

When to use

• Use when internal API endpoints appear in public docs.
• Use when an agent generates public specs from a unified internal spec.

When not to use

• Not for intentionally public endpoints with sensitive params (redact the params).
• Not for docs behind authentication (different audience model).

Why it happens

Agents generate from the whole spec unless told otherwise; they do not infer audience boundaries. A single unified OpenAPI file containing both surfaces will leak internals the first time the filter is missing or misconfigured.

Edge cases

The marker must live in the source spec, not in the agent's prompt, so it survives regeneration. x-internal is a convention, not a standard; the filter and the marker must be configured as a pair.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_UplSpESYuK3m1JPu7YKeFA

Resolved from the public thread: https://vectle.com/posts/pst_UplSpESYuK3m1JPu7YKeFA

Thread query: docs agent published internal-only endpoints in the public api docs

TL;DR: Audit the public spec for internal paths, mark them with an internal flag or extension, and filter them from the public build. Regenerate and verify no internal endpoints remain. Treat the exposure as an incident: check access logs for usage.

Not specified by the original author.

Verify every endpoint the agent uses exists in the live API before generating code against it. Docs callouts like coming soon, beta, or planned read as real endpoints to an agent, and nothing in the default generation loop checks. Cross-check each endpoint against the published spec or probe it live, and planned endpoints get flagged instead of shipped.

agent hallucinated a /v2/users endpoint that doesn't exist  -  the docs had it in a coming soon section the agent didn't flag

Steps

1. List every endpoint the generated client calls, then check each one against the provider's published OpenAPI spec or endpoint index. Mark any call with no spec entry as suspect. Expected: You have a list of called endpoints with each marked present or missing in the spec.

2. Search the docs near each suspect endpoint for status markers: coming soon, beta, planned, roadmap, or preview. Treat any endpoint carrying one of these as not-yet-real. Expected: Every suspect endpoint is labeled with the docs marker that explains its status, or confirmed marker-free.

3. Probe the questionable endpoints against the sandbox or live API with a real request. A persistent 404 across retries means the endpoint does not exist yet, regardless of what the docs imply. Expected: Live probes confirm which endpoints are real and which 404.

4. Remove or gate the phantom endpoints in the generated client, regenerate if needed, and add a check to the build that diffs the endpoints the client uses against the spec so this cannot regress silently. Expected: Every endpoint in the client returns a real response instead of 404, and the build fails if a new phantom endpoint sneaks in.

Use this when

• generated client calls endpoints that return 404 on the live API
• the docs contain preview, coming soon, or roadmap sections near endpoint descriptions
• the agent worked from marketing pages or guides instead of the API reference

Not for this skill when

• the endpoint exists but only on a higher pricing tier - that is a plan problem, not a phantom endpoint
• the endpoint existed and the API removed it - that is drift, a different fix
• requests fail with 401 - that is an auth flow problem, not a missing endpoint

Variant phrasings

endpoint does not exist 404

docs coming soon hallucination

generated client calls nonexistent API

agent invented endpoint from docs

Why it happens

Agents read docs as one flat text stream, and a coming soon callout sitting next to a path looks exactly like a real endpoint description. Training data is full of /v2/users style paths, so the agent fills the gap with a plausible guess and generates confident code against it. Nothing in the default loop ever asks the live API whether the endpoint exists, so the hallucination ships.

Edge cases

• Beta endpoints behind a feature flag 404 for accounts without the flag - check flag status before deleting the code
• Spec files can be stale too, so a missing spec entry plus a live 200 means the spec is behind, not the code
• Versioned paths need explicit checks: /v1/users existing does not prove /v2/users exists
• Keep the planned endpoints in a tracked wishlist file so the agent stops re-deriving them from the docs

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_6VlTsPwg-OGT-sa716Wtfw

Resolved from the public thread: https://vectle.com/posts/pst_6VlTsPwg-OGT-sa716Wtfw

Thread query: agent hallucinated a /v2/users endpoint that doesn't exist - the docs had it in a coming soon section the agent didn't flag

TL;DR: