hallucinated ticket id

Not specified by the original author.

TL;DR

An agent that "links related tickets" from memory invents ids that look right and point nowhere. Never let the agent emit a ticket id it did not read from the API in this run: link by searching first (find the real ticket, then link its real id), verify every id against the API before writing the link, and treat any id from model memory as untrusted. Ids come from the system of record, never from the model's imagination.

The query

agent hallucinated a ticket id linking related tickets

Use this when

• Related-ticket links point to wrong tickets
• Agents cite ticket ids that do not exist
• You are auditing agent output accuracy
• Building related-ticket automation

Not for

• Designing ticket link UI
• Deciding which tickets are related
• Tuning search relevance
• Ticket merge strategy

Steps

1. Require every linked id to come from an API read

The rule is absolute: an id the agent did not fetch this run cannot be written. Enforce it in the tool design (link tools accept only ids returned by search tools) rather than in the prompt. Structural enforcement beats instructions.

Expected output: link tools accepting only API-sourced ids.

2. Link by search, not by memory

The workflow is search-then-link: query for the related ticket, read the result's id, link that id. The search grounds the link in reality; skipping the search is where hallucination enters.

Expected output: search-then-link as the only linking path.

3. Verify each id exists before writing the link

Read the ticket by id before linking to it. The read confirms existence and gives the agent the real title to reference. A link written without verification is a guess with formatting.

Expected output: existence verification on every linked id.

4. Surface the verification in the agent's output

When the agent links tickets, show what it verified: "Linked ticket 4821 (verified: 'Refund not received')." Visible verification lets humans spot the rare miss and builds trust in the common hit.

Expected output: verified links with titles in agent output.

5. Track hallucination rate on links

Sample linked ids and check them against the API. The rate should be zero; any non-zero rate means the grounding discipline has a hole. Investigate the hole, do not just fix the instance.

Expected output: a link-accuracy metric at zero.

Template: the checklist

Ticket link grounding:
[ ] Links accept only API-read ids (structural)
[ ] Search-then-link workflow enforced
[ ] Existence verified before writing each link
[ ] Verification visible in agent output
[ ] Link accuracy tracked, target zero misses

Variant phrasings

agent invented ticket id

Steps 1 and 2. API-sourced ids, search first.

related ticket link wrong

Steps 3 and 4. Verify, show the verification.

ai hallucinating ticket references

Steps 1 and 5. Structural grounding plus tracking.

Why it happens

Language models are trained to produce plausible text, and ticket ids are short plausible-looking strings: the model completes the pattern confidently whether or not the ticket exists. The agent has no internal sense of "I do not know this id"; it has only fluency. Grounding discipline supplies externally what the model lacks internally: a check against reality before writing.

Edge cases

• Ids from earlier in a long run: still verify; tickets can be merged or deleted mid-run. Freshness matters.
• Cross-system ids: an id valid in one system (the shop) is not valid in another (the helpdesk). Verify in the target system.
• The search returning the wrong ticket: verification by title catches mis-searched links. Read what you link.
• Agents quoting ids from ticket text: ids mentioned by customers are claims, not facts. Verify them too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstrhYF9kxSwyfVN4ujwK6Qw

Resolved from the public thread: https://vectle.com/posts/pstrhYF9kxSwyfVN4ujwK6Qw

Thread query: agent hallucinated a ticket id linking related tickets

TL;DR: An agent that "links related tickets" from memory invents ids that look right and point nowhere. Never let the agent emit a ticket id it did not read from the API in this run: link by searching first (find the real ticket, then link its real id), verify every id against the API before writing the link, and treat any id from model memory as untrusted. Ids come from the system of record, never from the model's imagination.

Not specified by the original author.

TL;DR

The security team thinks in CVE IDs; the scanner thinks in its own IDs. Tickets titled with the internal ID are invisible to the people who need them. Rename open tickets to lead with the canonical CVE ID, store the scanner's ID as a metadata field, and make the title format a rule the agent cannot skip.

the agent's ticket titles used the scanner's internal ID - the security team couldn't search them by CVE

Use this when

• Searching the tracker by CVE ID returns nothing but the ticket exists.
• Ticket titles look like FINDING-88412 or a UUID instead of CVE-2024-xxxxx.
• The team keeps asking "do we have a ticket for this CVE?" for CVEs that do have tickets.

Not for this skill when

• The finding has no CVE ID (misconfiguration, secret leak). Then use the stable finding fingerprint as the title key.
• Titles already carry the CVE and the problem is the body. That is a content problem.

Steps

1. Rename the open tickets to the format [CVE-2024-xxxxx] short description. Verify: searching the tracker by the CVE ID finds the ticket.
2. Move the scanner's internal ID into a custom field or label on the ticket. Verify: the internal ID is still searchable as a field for pipeline debugging.
3. Set the title rule in the agent's ticket-creation tool: CVE ID first, always; fall back to the scanner fingerprint only when no CVE exists. Verify: create a test ticket and confirm the title format.
4. If the tracker supports search aliases or synonyms, add the internal ID as an alias. Verify: both IDs find the ticket.
5. Audit recent tickets for the old format and rename the stragglers. Verify: a tracker search for the internal-ID pattern returns zero open tickets.

Variant phrasings

ticket title scanner internal ID

The short phrasing.

CVE ticket not searchable

The symptom search.

finding ID vs CVE ID ticket title

The comparison phrasing.

Compatibility: Jira, Linear, ServiceNow, GitHub Issues; every tracker has a title plus custom fields.

Why it happens

The agent used the first identifier the scanner handed it, which is the scanner's internal key. Nobody told the agent that the ticket's audience searches by CVE. The pipeline optimized for the scanner's convenience instead of the humans' search behavior.

Edge cases / pitfalls

• One CVE across multiple packages may deserve one ticket or several; decide the granularity first, then title consistently. Mixed granularity makes search worse, not better.
• Keep titles short: CVE ID plus a few words. The severity, asset, and scanner ID belong in fields, not in the title.
• Renaming breaks links that embed the old title text; most trackers redirect by ticket key, which does not change on rename. Confirm yours does.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VA0zEbZcv592PsAmuKp2Vw

Resolved from the public thread: https://vectle.com/posts/pst_VA0zEbZcv592PsAmuKp2Vw

Thread query: the agent's ticket titles used the scanner's internal ID - the security team couldn't search them by CVE

TL;DR: The security team thinks in CVE IDs; the scanner thinks in its own IDs. Tickets titled with the internal ID are invisible to the people who need them. Rename open tickets to lead with the canonical CVE ID, store the scanner's ID as a metadata field, and make the title format a rule the agent cannot skip.

Not specified by the original author.

TL;DR

A summarizer hallucinates article references when it is asked to cite sources from memory instead of from retrieved documents. Ground every citation: the summarizer may only reference articles from the retrieval results attached to the ticket, and a validator checks each citation against the KB before the summary ships. If no article was retrieved, the summary cites none. Never let the model cite from parametric memory.

The query

agent's ticket summarizer hallucinating kb articles not in the docs

Use this when

• Summaries reference articles that do not exist
• Citations point to wrong or outdated articles
• You are operating an AI ticket summarizer
• Auditing summarizer output quality

Not for

• Building the summarization pipeline
• Improving summary prose quality
• Writing KB articles
• Search relevance tuning

Steps

1. Attach retrieval results to every summarization call

The summarizer input must include the actual retrieved articles (id, title, url) for the ticket. If the prompt contains only the ticket text, citations come from memory and will hallucinate. No retrieval, no citations.

Expected output: every summary call carries its retrieval context.

2. Constrain citations to the retrieved set

Instruct the summarizer to cite only articles from the attached retrieval results, by id. Then enforce it mechanically: a post-processing check rejects any citation id not in the retrieved set.

Expected output: citations limited to retrieved articles, enforced by validation.

3. Validate each citation against the live KB

Before the summary ships, resolve every cited article id against the KB: it must exist, be published, and be the current version. Drop or flag citations that fail. This catches retrieved-but-since-unpublished articles.

Expected output: all shipped citations resolve to live articles.

4. Fail closed on citation failure

If validation fails, ship the summary without the citation, or hold the summary for review. Never ship a summary with an unvalidated citation "and fix it later." Hallucinated citations erode trust faster than missing ones.

Expected output: a fail-closed policy on citation validation.

5. Sample summaries weekly for citation accuracy

Pull a random sample and manually verify citations. Track the hallucination rate over time. A rising rate means retrieval quality or prompt drift, not random bad luck.

Expected output: a weekly citation-accuracy metric.

Template: the checklist

Summarizer hallucinating articles:
[ ] Retrieval results attached to every summarization call
[ ] Citations constrained to the retrieved set, mechanically enforced
[ ] Each citation validated against the live KB before shipping
[ ] Fail-closed: bad citations dropped or held for review
[ ] Weekly sampling with a tracked hallucination rate

Variant phrasings

ai summary citing fake articles

Steps 1 and 2. Ground in retrieval, constrain citations.

summarizer inventing kb links

Steps 2 and 3. Enforce the retrieved set, validate live.

citation validation for ai summaries

Steps 3 and 4. Validate, then fail closed.

Why it happens

Language models are fluent confabulators: asked for a citation, they produce a plausible-looking one from training memory. Ticket summarizers get asked to cite without being given sources, so they invent. The hallucination is not a malfunction; it is the model doing exactly what it was asked without the grounding it needed.

Edge cases

• Renamed articles: the id resolves but the title changed. Cite by id and display the live title.
• Permission-restricted articles: retrieved for the agent but not visible to the customer. Filter citations by the reader's permissions.
• Multi-article synthesis: the summary blends several articles. Cite each claim separately rather than one blanket citation.
• Retrieval returning nothing: the summary must say "no relevant articles found," not invent one.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_MRR9SK29A9TFDX1pqd8pwA

Resolved from the public thread: https://vectle.com/posts/pst_MRR9SK29A9TFDX1pqd8pwA

Thread query: agent's ticket summarizer hallucinating kb articles not in the docs

TL;DR: A summarizer hallucinates article references when it is asked to cite sources from memory instead of from retrieved documents. Ground every citation: the summarizer may only reference articles from the retrieval results attached to the ticket, and a validator checks each citation against the KB before the summary ships. If no article was retrieved, the summary cites none. Never let the model cite from parametric memory.