agent hallucinated a /v2/users endpoint that doesn't exist - the docs had it in a coming soon section the agent...

Verify every endpoint the agent uses exists in the live API before generating code against it. Docs callouts like coming soon, beta, or planned read as real endpoints to an agent, and nothing in the default generation loop checks. Cross-check each endpoint against the published spec or probe it live, and planned endpoints get flagged instead of shipped.

agent hallucinated a /v2/users endpoint that doesn't exist  -  the docs had it in a coming soon section the agent didn't flag

Steps

1. List every endpoint the generated client calls, then check each one against the provider's published OpenAPI spec or endpoint index. Mark any call with no spec entry as suspect.

Expected: You have a list of called endpoints with each marked present or missing in the spec.

1. Search the docs near each suspect endpoint for status markers: coming soon, beta, planned, roadmap, or preview. Treat any endpoint carrying one of these as not-yet-real.

Expected: Every suspect endpoint is labeled with the docs marker that explains its status, or confirmed marker-free.

1. Probe the questionable endpoints against the sandbox or live API with a real request. A persistent 404 across retries means the endpoint does not exist yet, regardless of what the docs imply.

Expected: Live probes confirm which endpoints are real and which 404.

1. Remove or gate the phantom endpoints in the generated client, regenerate if needed, and add a check to the build that diffs the endpoints the client uses against the spec so this cannot regress silently.

Expected: Every endpoint in the client returns a real response instead of 404, and the build fails if a new phantom endpoint sneaks in.

Use this when

• generated client calls endpoints that return 404 on the live API
• the docs contain preview, coming soon, or roadmap sections near endpoint descriptions
• the agent worked from marketing pages or guides instead of the API reference

Not for this skill when

• the endpoint exists but only on a higher pricing tier - that is a plan problem, not a phantom endpoint
• the endpoint existed and the API removed it - that is drift, a different fix
• requests fail with 401 - that is an auth flow problem, not a missing endpoint

Variant phrasings

endpoint does not exist 404

docs coming soon hallucination

generated client calls nonexistent API

agent invented endpoint from docs

Why it happens

Agents read docs as one flat text stream, and a coming soon callout sitting next to a path looks exactly like a real endpoint description. Training data is full of /v2/users style paths, so the agent fills the gap with a plausible guess and generates confident code against it. Nothing in the default loop ever asks the live API whether the endpoint exists, so the hallucination ships.

Edge cases

• Beta endpoints behind a feature flag 404 for accounts without the flag - check flag status before deleting the code
• Spec files can be stale too, so a missing spec entry plus a live 200 means the spec is behind, not the code
• Versioned paths need explicit checks: /v1/users existing does not prove /v2/users exists
• Keep the planned endpoints in a tracked wishlist file so the agent stops re-deriving them from the docs

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_6VlTsPwg-OGT-sa716Wtfw