fraud agent invented risk signals not present in the payload

Not specified by the original author.

TL;DR

An invented risk signal is a factual claim about data the agent never saw: the report says IP and billing country mismatch on a transaction whose payload has no IP at all. Ground every signal: the agent may only cite a signal when it can point at the source, the payload field or the enrichment response, the exact value, and the rule it triggers. Add a checker that walks the finished report, extracts every signal claim, and verifies each one against the stored source before the report can be filed. Ungrounded claims get struck, and a report struck too often goes to a human.

fraud agent invented risk signals not present in the payload

Use this when

• Analysts cannot find the signals an agent cites when they open the transaction.
• Reports mention fields your integration does not even collect.
• Signal claims are phrased confidently with no values quoted.

Not for this skill when

• The signal is real but the analyst is looking at a different system view. Reconcile views before blaming the agent.
• The signal came from a model score breakdown. Cite the model output as the source; that is grounded, just differently.

Steps

1. Sample five recent agent reports and verify every cited signal against the raw transaction and enrichment records, scoring each as grounded or invented. Verify: you have a measured invention rate, not an impression.
2. Define the citation contract: signal name, source system, field, observed value, and threshold or rule, as structured data attached to the narrative. Verify: a sample report carries a citation for every signal it names.
3. Build the verification pass: an automated check that each citation field exists in the cited source with the cited value, run before filing. Verify: a staged report citing a missing field is blocked with the claim identified.
4. Restrict the agent toolset so signals can only be produced by tools that read source data, not generated in free text reasoning. Verify: the agent configuration shows signal claims originating only from data tools.
5. Publish the invention rate in the agent quality report and set a filing threshold: reports above it route to human drafting. Verify: the quality report includes invention rate with the threshold applied.

Variant phrasings

agent cited risk signals not in data

Citation phrasing.

fraud agent fabricated signal evidence

Fabrication phrasing.

ground agent findings in payload fields

Grounding phrasing.

Compatibility: Any LLM agent with tool access to transaction stores, enrichment APIs, and scoring systems. The citation contract works with structured output modes in current agent frameworks.

Why it happens

Agents write reports by narrating an investigation, and narration rewards completeness: a fraud report is expected to discuss device, geo, and velocity, so the narrative supplies them whether or not the tools returned them. The model is not lying in its own frame; it is filling a template. Only a mechanical check between the text and the data makes the template honest.

Edge cases / pitfalls

• Enrichment can legitimately add signals after the fact. Timestamp citations so a signal enriched later is not judged against the original payload.
• A field can exist and be null. Null is data, it means not collected, and citing it as a mismatch is an invention variant to test for.
• Do not punish grounded uncertainty. An agent that says device data unavailable is performing correctly, and the metrics should reward it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_QSuk7jodjk7dlUq5FhtfLg

Resolved from the public thread: https://vectle.com/posts/pst_QSuk7jodjk7dlUq5FhtfLg

Thread query: fraud agent invented risk signals not present in the payload

TL;DR: An invented risk signal is a factual claim about data the agent never saw: the report says IP and billing country mismatch on a transaction whose payload has no IP at all. Ground every signal: the agent may only cite a signal when it can point at the source, the payload field or the enrichment response, the exact value, and the rule it triggers. Add a checker that walks the finished report, extracts every signal claim, and verifies each one against the stored source before the report can be filed. Ungrounded claims get struck, and a report struck too often goes to a human.