fraud agent hallucinated a fraud ring with no linked transactions

TL;DR

A fraud ring is a graph claim: these transactions are connected by shared cards, devices, addresses, accounts, or IPs. An agent hallucinates a ring when it asserts the connections in prose without any shared identifier existing in the data. Make the agent prove the graph: a ring can only be created from an edge list where each edge names the identifier type, the identifier value fingerprint, and the two transactions it connects, all pulled from source payloads. If the edge list is empty, there is no ring, only a cluster of suspicious transactions, and the output must say exactly that.

fraud agent hallucinated a fraud ring with no linked transactions

Use this when

• An agent reported a ring and analysts found no links between its transactions.
• Ring reports use confident language with no identifier evidence attached.
• Ring counts in reporting exceed what analysts confirm.

Not for this skill when

• Transactions share identifiers but analysts judge them benign, for example a corporate NAT IP. That is link interpretation, not hallucination.
• The agent missed a real ring. Recall problems need better linking features, not evidence gates.

Steps

1. Take the reported ring and demand its edge list: for each claimed connection, the identifier and the transactions it joins, checked against the source data. Verify: each edge is either verified in source data or marked invented.
2. Change the agent output contract: ring findings must include a machine readable edge list built only from fields present in the transaction payloads, with no edge allowed from narrative inference. Verify: a staged ring with no shared identifiers is output as unlinked suspicious transactions instead.
3. Add a verifier step that replays the edge list against the source store and rejects any edge whose identifier does not appear in both transactions. Verify: an invented edge in testing is caught by the verifier with the bad edge named.
4. Separate suspicion from connection in the UI and reports: clusters, rings, and confirmed rings are three distinct states with distinct counts. Verify: the dashboard no longer counts unverified clusters as rings.
5. Sample agent ring reports weekly for analyst confirmation and track the confirmed rate as the agent ring quality metric. Verify: the weekly confirmed rate is reported and rising after the gate.

Variant phrasings

agent invented fraud ring no evidence

Evidence phrasing.

hallucinated ring linked transactions missing

Link phrasing.

require shared identifier fraud ring detection

Requirement phrasing.

Compatibility: Any LLM or rules based fraud investigation agent with access to transaction, device, and account stores. The edge list contract maps to graph tools and case management systems.

Why it happens

Language models complete patterns: several suspicious transactions in one report invite a unifying story, and a ring is the most available story in fraud. Nothing in a prose only workflow forces the model to check that a single shared identifier actually exists, so fluency manufactures the link. The fix is structural, evidence as a required artifact, not a sterner prompt.

Edge cases / pitfalls

• Shared identifiers can be coincidental, a big employer IP or a reshipper address. Verified edges prove connection, not guilt; analysts still judge the ring.
• Fingerprint identifiers before putting them in reports. Ring evidence does not need raw card or device values to be checkable.
• Keep the rejected ring reports. They are the training signal for where the agent pattern completes, and deleting them hides the failure rate.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2zNJLNYWmukuDxImTTmwqw