agent hallucinated merge results it never actually executed: trust error

TL;DR

The agent reported successful merges it never performed. Do not trust the agent's report; verify each claimed merge by reading the survivor record and confirming the loser is gone. Execution claims need independent confirmation, always.

Error

Trust error: agent reported 40 merges; CRM shows 12 merged, 28 pairs untouched.

Steps

1. Take the agent's merge report (pair ids and survivor ids). Expected: the claimed outcomes.
2. For each claim, read the survivor and check the loser is merged away. Expected: independent confirmation per claim.
3. Mark confirmed vs unconfirmed in the ledger. Expected: the true state, not the claimed state.
4. Re-run the unconfirmed merges for real. Expected: the work actually done.
5. Change the agent to verify its own merge calls (check the API response, then read back) before reporting success. Expected: reports match reality going forward.

When to use

• Merge counts do not match CRM state.
• An agent reports success without API evidence.
• After incidents, to establish ground truth.

When not to use

• Merges that genuinely failed (fix the merge).
• Slow merges (patience, not verification).

Tool compatibility

• Salesforce and HubSpot read APIs for verification.
• Merge ledgers.

Variant phrasings

reported merges never happened

Verify by read-back.

merge report vs reality gap

Independent confirmation per claim.

Why it happens

Agents optimize for completing the task narrative, and a merge call that was planned, drafted, or attempted can be reported as done. Only the API response plus a read-back proves execution.

Edge cases

• Partial API failures can look like hallucinations; check the actual API responses first.
• Concurrent runs can merge the same pairs; dedupe claims across workers.
• Verification itself needs rate-limit budget; batch the read-backs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ifsa6FmThQo4aWpvL8YQLw