agent hallucinated sso configured before saml metadata uploaded

TL;DR

Your agent told everyone SSO was configured when the SAML metadata was never uploaded: it hallucinated the completion of a step it never performed. The fix: make every setup step verify its own result against the identity provider before reporting success. No verification, no "done." Add a pre-flight checklist so the agent cannot reach the "configured" state without the metadata upload succeeding first.

The error

Agent: "SSO configured successfully for [workspace]"
IdP admin console: no SAML app, no metadata, nothing configured

Steps

1. Confirm the hallucination. Check the IdP console for the SAML app the agent claims to have created. Expected output: nothing there; the step never happened.
2. Find where the agent skipped verification. Read the agent's SSO routine and find the step that reports success without reading anything back. Expected output: one step that assumes success (for example, "metadata uploaded" with no confirmation).
3. Add verify-after-write to every step. After uploading metadata, fetch the IdP's app config and confirm the metadata is present. Expected output: the step fails loudly when the upload fails.
4. Gate "configured" on the full checklist. The agent may only report SSO configured after each checklist item verifies. Expected output: partial setups report as partial, never as done.
5. Run a clean SSO setup end to end. Expected output: the agent's report matches the IdP console exactly.

When to use

• The agent says SSO is done but the IdP shows nothing
• SSO "worked" in testing but real setups are incomplete
• The agent's success reports do not match any system of record

When not to use

• The IdP returns a real SAML error: debug the metadata, certificates, or attribute mapping
• SSO works for some users but not others: an attribute or group mapping problem
• You are setting up SSO manually: this is about the agent's verification gap

Tool compatibility

SAML SSO setup against any identity provider (Okta, Entra ID, Google Workspace, Auth0) driven by an agent framework. The verify-after-write pattern applies to any multi-step configuration the agent performs.

Variant phrasings

agent says sso done but not configured

Add verification reads after every write.

sso setup incomplete agent reported success

Gate "done" on a verified checklist.

saml metadata never uploaded

The agent skipped the step and reported success anyway.

onboarding agent hallucinated completion

Trust the system of record, not the agent's log.

Why it happens

Language models are completion engines: asked to set up SSO, they produce the narrative of a successful setup, including steps they never executed. Tool-using agents are supposed to ground that narrative in tool results, but if a step has no verification read, there is nothing to contradict the story. The metadata upload is the classic victim because it is a fiddly file-handling step that fails quietly. The structural fix is verify-after-write: no step counts until the system of record confirms it.

Edge cases

• The verification read is cached: a stale read can confirm a write that never landed. Read with cache bypass where the API allows it.
• The agent verifies the wrong app: scope every check to the exact app ID created in this run.
• Partial IdP APIs: some providers do not expose the metadata back. Then verify indirectly (a test login) instead of skipping verification.
• The human uploads the metadata out of band: the agent should detect the existing config and adopt it, not overwrite it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DvRVdeHekpTeDtURIY7WoA