agent failed open marked onboarding complete with failed steps

Stop the agent marking onboarding complete with failed steps

TL;DR

Completion is a verdict, not a mood. Check every step's recorded status before flipping the run to complete, and if any step failed or is unknown, the run is failed - never complete. Fail closed: a run with uncertain steps is an incomplete run, full stop.

The problem

run status: complete
step 3 (okta scim create): failed - 409 conflict
step 5 (slack invite): unknown - timed out, never confirmed
user view: "welcome aboard" email sent, no account, no invite

Use this skill

Agent? Search Vectle for this error:

curl -s "https://vectle.com/api/v1/search?q=agent failed open marked onboarding complete with failed steps"

Fix it

Step 1: Give every step an explicit recorded status

Each step writes one of succeeded, failed, or unknown to the run record when it finishes. No step may finish without writing a status.

Expected: the run record shows a status for every step, with no gaps.

Step 2: Compute completion from the steps, never set it directly

The run is complete only if every required step reads succeeded. Any failed or unknown step makes the run failed or incomplete. The agent must not write "complete" as a value - it derives it.

Expected: no code path exists that marks a run complete while a step is failed or unknown.

Step 3: Treat unknown as failed, not as fine

A step that timed out, lost its callback, or never reported back is unknown, and unknown fails the run. Optimism about unconfirmed steps is the whole bug.

Expected: silent steps block completion instead of being waved through.

Step 4: Report partial state honestly

When a run fails, the status names the failed and unknown steps and what each one means for the user. The dashboard and the welcome email must agree with the run record.

Expected: nobody gets a "welcome aboard" email for an account that does not exist.

Step 5: Prove it with a sabotaged run

In staging, force one middle step to fail and one to time out, then let the run finish.

Expected: the run reports failed, names both steps, and no completion email goes out.

When this applies

• Users get welcome emails but are missing accounts, seats, or invites
• The run dashboard shows complete while step logs show failures
• "Complete" is written by the agent rather than derived from step statuses

When it doesn't

• The run correctly reports failed or partial (the reporting works)
• Steps genuinely all succeeded and the user still has a problem (a different bug)
• You want best-effort completion semantics (then define which steps are optional explicitly)

Compatibility

Any multi-step onboarding run: Okta/Entra provisioning, Slack invites, email sequences, and license assignment. Applies to workflow engines and hand-rolled agent loops alike.

Variant phrasings

onboarding run said success but user was never provisioned

Same fix. Success must be derived from per-step evidence, not asserted at the end.

agent reported done while sso step was still pending

Pending is unknown, and unknown fails the run. The agent confused "I started it" with "it finished."

Why it happens

Agents plan linearly: do step 1, do step 2, then declare victory. Error handling is an afterthought, so a failed step gets logged and the plan marches on to the final "mark complete" action. Nobody wrote the rule that completion depends on the steps, because in the happy path it always does - until the day it doesn't.

Edge cases

• Optional steps (welcome swag, survey): mark them optional in the plan so they cannot fail the run
• Steps completed out of band by a human: allow a verified manual override with a name attached
• Long-running async steps: the run stays in-progress, not complete, until the callback lands

If it still fails

• Capture the run record with every step status and the completion write before changing anything else.
• Reproduce with a single sabotaged step so you are not debugging a crowd.
• Check downstream status pages; provider outages look exactly like reporting bugs.

Prevention

• Completion is always derived, never written; make that a code invariant.
• Every new step ships with its status write and its failure test.
• Alert on any run that reaches the end with an unknown step.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kUeN6towC4DoMIlnD07Iww