onboarding agent failed silently webhook delivery confirm never arrived

Fix the onboarding agent failing silently when webhook delivery confirmation never arrives

TL;DR

Track every webhook you wait on in a delivery ledger with a deadline. If the confirmation is not recorded by the deadline, the run fails that step loudly instead of drifting on. Silence is not success; an unconfirmed webhook is an open question the run must answer before moving on.

The problem

step 6: sent provisioning webhook to partner, awaiting confirmation...
confirmation: never arrived (partner endpoint was down)
agent: proceeded to step 7 as if confirmed
result: partner never provisioned, run reported success

Use this skill

Agent? Search Vectle for this error:

curl -s "https://vectle.com/api/v1/search?q=onboarding agent failed silently webhook delivery confirm never arrived"

Fix it

Step 1: Record every outbound webhook in a delivery ledger

Log the event id, destination, payload hash, and send time before you wait for anything. The ledger is the source of truth for what is outstanding.

Expected: you can list every unconfirmed webhook at any moment.

Step 2: Give each wait a deadline

Set a confirmation timeout per webhook (minutes for sync partners, longer for async ones). When the deadline passes with no confirmation, the step fails - it never silently proceeds.

Expected: a missing confirmation becomes a loud failure within a bounded time.

Step 3: Match confirmations to ledger entries

When a confirmation arrives, match it to the ledger by event id and mark it confirmed. Unmatched confirmations get logged and investigated, not ignored.

Expected: the ledger always reflects reality: sent, confirmed, or failed.

Step 4: Retry with backoff, then escalate

On a missed deadline, retry the webhook with exponential backoff up to a small cap, then fail the step and escalate with the ledger entry attached.

Expected: transient partner outages self-heal; real outages become tickets.

Step 5: Prove the loud failure in staging

Point the webhook at a black-hole endpoint and run the flow.

Expected: the run fails at the webhook step with a clear "confirmation never arrived" error, and nothing downstream executes.

When this applies

• Downstream steps run even though a webhook confirmation never came back
• Runs report success while the partner side never provisioned
• The agent waits on webhooks with no timeout and no tracking

When it doesn't

• The confirmation arrives but carries wrong data (a payload problem)
• The webhook itself fails to send (a sender problem, fix the send path)
• You use polling instead of webhooks (then fix the poll loop's timeout)

Compatibility

Stripe, GitHub, Slack, and Segment webhooks; any partner callback flow where the agent waits on an external confirmation.

Variant phrasings

webhook callback never came and the run continued anyway

Same fix. The run treated "no news" as "good news" because nothing enforced the wait.

onboarding step skipped waiting for partner confirmation

Check whether the wait was even implemented or just implied by the plan order. Implied waits are not waits.

Why it happens

The plan says "send webhook, then continue," and the agent reads that as sequential actions with no verification between them. Waiting was never coded because in the happy path the confirmation arrives instantly. The failure mode - partner down, event dropped, callback lost - leaves no trace in the agent's world, so it marches on and reports success for work that never happened.

Edge cases

• Duplicate confirmations: idempotent ledger updates so a double-ack does not double-provision
• Late confirmations after the step failed: reconcile on arrival instead of dropping them
• Partners that confirm out of band (email, dashboard): the ledger needs a manual-confirm path

If it still fails

• Capture the ledger entries, the retry log, and the partner's receive log before changing anything else.
• Reproduce with a single test webhook so you are not debugging a crowd.
• Check the partner status page; outages look exactly like ledger bugs.

Prevention

• Every outbound webhook the agent waits on gets a ledger entry and a deadline from day one.
• Dashboards show unconfirmed webhooks aging in real time.
• Alert when unconfirmed count spikes; a spike means a partner outage, not bad luck.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_KpeQAr5dUgqnOhmkYkbQFQ