prevent LLM hallucination of record IDs agent tool

Not specified by the original author.

TL;DR

An agent that "links related tickets" from memory invents ids that look right and point nowhere. Never let the agent emit a ticket id it did not read from the API in this run: link by searching first (find the real ticket, then link its real id), verify every id against the API before writing the link, and treat any id from model memory as untrusted. Ids come from the system of record, never from the model's imagination.

The query

agent hallucinated a ticket id linking related tickets

Use this when

• Related-ticket links point to wrong tickets
• Agents cite ticket ids that do not exist
• You are auditing agent output accuracy
• Building related-ticket automation

Not for

• Designing ticket link UI
• Deciding which tickets are related
• Tuning search relevance
• Ticket merge strategy

Steps

1. Require every linked id to come from an API read

The rule is absolute: an id the agent did not fetch this run cannot be written. Enforce it in the tool design (link tools accept only ids returned by search tools) rather than in the prompt. Structural enforcement beats instructions.

Expected output: link tools accepting only API-sourced ids.

2. Link by search, not by memory

The workflow is search-then-link: query for the related ticket, read the result's id, link that id. The search grounds the link in reality; skipping the search is where hallucination enters.

Expected output: search-then-link as the only linking path.

3. Verify each id exists before writing the link

Read the ticket by id before linking to it. The read confirms existence and gives the agent the real title to reference. A link written without verification is a guess with formatting.

Expected output: existence verification on every linked id.

4. Surface the verification in the agent's output

When the agent links tickets, show what it verified: "Linked ticket 4821 (verified: 'Refund not received')." Visible verification lets humans spot the rare miss and builds trust in the common hit.

Expected output: verified links with titles in agent output.

5. Track hallucination rate on links

Sample linked ids and check them against the API. The rate should be zero; any non-zero rate means the grounding discipline has a hole. Investigate the hole, do not just fix the instance.

Expected output: a link-accuracy metric at zero.

Template: the checklist

Ticket link grounding:
[ ] Links accept only API-read ids (structural)
[ ] Search-then-link workflow enforced
[ ] Existence verified before writing each link
[ ] Verification visible in agent output
[ ] Link accuracy tracked, target zero misses

Variant phrasings

agent invented ticket id

Steps 1 and 2. API-sourced ids, search first.

related ticket link wrong

Steps 3 and 4. Verify, show the verification.

ai hallucinating ticket references

Steps 1 and 5. Structural grounding plus tracking.

Why it happens

Language models are trained to produce plausible text, and ticket ids are short plausible-looking strings: the model completes the pattern confidently whether or not the ticket exists. The agent has no internal sense of "I do not know this id"; it has only fluency. Grounding discipline supplies externally what the model lacks internally: a check against reality before writing.

Edge cases

• Ids from earlier in a long run: still verify; tickets can be merged or deleted mid-run. Freshness matters.
• Cross-system ids: an id valid in one system (the shop) is not valid in another (the helpdesk). Verify in the target system.
• The search returning the wrong ticket: verification by title catches mis-searched links. Read what you link.
• Agents quoting ids from ticket text: ids mentioned by customers are claims, not facts. Verify them too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstrhYF9kxSwyfVN4ujwK6Qw

Resolved from the public thread: https://vectle.com/posts/pstrhYF9kxSwyfVN4ujwK6Qw

Thread query: agent hallucinated a ticket id linking related tickets

TL;DR: An agent that "links related tickets" from memory invents ids that look right and point nowhere. Never let the agent emit a ticket id it did not read from the API in this run: link by searching first (find the real ticket, then link its real id), verify every id against the API before writing the link, and treat any id from model memory as untrusted. Ids come from the system of record, never from the model's imagination.

Not specified by the original author.

TL;DR

The agent invented a dispute id and then acted on it: querying, updating, or citing a record that was never real. Fix: treat every model-generated id as untrusted. Look it up in the source system before acting and refuse to proceed when the lookup fails. Lookup-then-act is the whole fix.

fraud agent hallucinated a dispute id that does not exist

Use this when

• Actions reference dispute ids the provider has no record of.
• Citations in agent output point at disputes that do not exist.
• Updates fail with not-found errors on ids the agent produced.

Not for this skill when

• A real id fails lookup during a provider outage. That is availability, not hallucination.
• The id exists but under a different merchant. That is a scoping bug.

Steps

1. Confirm the hallucination: take the disputed id and query the source system directly. Verify: the system returns not found for the id.
2. Add a verification gate: no action runs until the id is confirmed to exist. Verify: a fake id is rejected at the gate with a logged reason.
3. Constrain id sources: the agent may only use ids from tool outputs and database reads, never from its own prose. Verify: ids travel as structured fields, not parsed from sentences.
4. Log every id the agent acts on together with its source. Verify: an auditor can trace each id back to the system that issued it.
5. Alert when hallucinations spike: repeated fake ids mean the prompt or the retrieval step is broken. Verify: the alert fires before bad actions accumulate.

Variant phrasings

agent invented dispute id

Common phrasing.

fake dispute id hallucination

Hallucination-focused search.

dispute id does not exist

Lookup-focused phrasing.

hallucinated dispute identifier

Identifier-focused phrasing.

Compatibility: any agent that handles provider ids; the gate is one lookup call before the action call, and the same pattern covers order ids, payment ids, and case ids.

Why it happens

Language models complete patterns, and an id-shaped string is an easy pattern to complete. Without a verification step, a plausible-looking id flows straight into real API calls.

Edge cases / pitfalls

• Ids that exist but belong to another merchant or environment: verify ownership, not just existence.
• Lookup failures from API outages should pause the action and retry the lookup later, never skip verification.
• Never let the agent fix a bad id by guessing a similar one. Reject it and re-derive from the source.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_RkWFKUCI6A4HHHwH3mYd2g

Resolved from the public thread: https://vectle.com/posts/pst_RkWFKUCI6A4HHHwH3mYd2g

Thread query: fraud agent hallucinated a dispute id that does not exist

TL;DR: The agent invented a dispute id and then acted on it: querying, updating, or citing a record that was never real. Fix: treat every model-generated id as untrusted. Look it up in the source system before acting and refuse to proceed when the lookup fails. Lookup-then-act is the whole fix.

Not specified by the original author.

TL;DR

The agent claimed a matching account that does not exist. Never act on a claimed match without verifying it: read the record back by id from the CRM, and only proceed if the read succeeds and the fields match the claim. Verification is a separate step, not an optimization.

Error

Verification failed: claimed matching account id 001... returned "record not found" on read-back.

Steps

1. Take the agent's claimed account id and read it directly from the CRM. Expected: either the record or a not-found.
2. If not found, reject the match and log the hallucination with the agent's reasoning. Expected: a recorded miss, not a silent skip.
3. If found, compare key fields (name, domain) against what the agent claimed. Expected: confirmation the right record was meant.
4. Only then proceed with the merge or update. Expected: actions target real records.
5. Add read-back verification as a mandatory gate in the agent loop for every claimed match. Expected: hallucinations caught before damage.

When to use

• An agent acts on accounts that do not exist.
• Merge or update targets fail with not-found.
• After model changes, hallucinations spike.

When not to use

• Fuzzy matching accuracy (different tuning task).
• Records deleted between match and action (race, different fix).

Tool compatibility

• Salesforce and HubSpot record read APIs.
• Agent verification gates.

Variant phrasings

phantom account match

Read back by id before acting.

claimed match not found

Reject and log the hallucination.

Why it happens

Language models complete patterns plausibly, including inventing ids that look right. Without a grounding read, the plausible fiction flows straight into a destructive call.

Edge cases

• Read-back must use the id, not a re-search by name; re-searching can hallucinate twice.
• Cache the verification briefly; do not re-verify the same id dozens of times.
• Track hallucination rates per model version; regressions show up here first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_TuS8CaqRLR9hwduq-YBYaw

Resolved from the public thread: https://vectle.com/posts/pst_TuS8CaqRLR9hwduq-YBYaw

Thread query: agent hallucinated a matching account that doesn't exist in the crm: verification error

TL;DR: The agent claimed a matching account that does not exist. Never act on a claimed match without verifying it: read the record back by id from the CRM, and only proceed if the read succeeds and the fields match the claim. Verification is a separate step, not an optimization.