agent hallucinating manager names that are not in HR data: generated wron

Not specified by the original author.

TL;DR

Constrain the agent to source values only and validate every invented-prone field against the system of record before it writes. The agent generated a manager who left the company because a required field was missing or ambiguous in its input and it filled the gap plausibly instead of stopping. Ground the step with a direct lookup, reject any value not traceable to the source, and make missing data a halt with a clear request, never a creative writing exercise.

agent hallucinating a manager who left the company: generated wrong values and validation failed

Use this when

• The agent output contains values that appear in no source record, such as invented assignments, people, or stock items.
• Failures happen on hires whose source records have missing or unusual fields the agent must interpret.
• Validation in the target rejects the generated values, or worse, occasionally accepts them.

Not for this skill when

• The source record itself holds the wrong value entered by a person. Correct the source data process instead.
• The agent copies a correct value into the wrong field. That is a mapping bug, not hallucination.

Steps

1. Capture the failing case completely: the agent input record, the exact values it generated, and its reasoning trace if available, and mark each generated value as present in the source, derived by a stated rule, or invented. Verify: every generated value is classified with evidence
2. Find the gap that invited invention: the missing, blank, or ambiguous source field the agent needed for this step, such as an absent assignment list, an unnamed approver, or an unchecked availability. Verify: the triggering gap is identified in the input record
3. Ground the step: give the agent a lookup tool against the authoritative list in Entra ID, for example the real assignment catalogue, people directory, or inventory, and require it to select only values the lookup returns. Free-text generation for these fields is disabled by the step design. Verify: the step can only emit values returned by the lookup in a test with a missing field
4. Add a validation gate before any write: each critical value is checked against the source record or the lookup result, and any mismatch or untraceable value halts the step with a specific message naming the field and the hire. Verify: the gate blocks a seeded invented value in testing
5. Change the missing-data behaviour in the agent instructions and step logic: when a required value is absent, the agent stops that hire, records the exact missing field, and queues the hire for a human data fix, then continues with other hires. Verify: a test hire with a missing field is queued with the field named, not invented around
6. Re-run the affected hires once their source gaps are fixed, confirm the assign Microsoft 365 licenses step uses only traceable values, and audit recent completed runs for silently accepted invented values that need correction in Entra ID. Verify: the audit finds no remaining untraceable values or lists them as owned cleanups

Why it happens

Language-model agents complete patterns: faced with a required field and no value, the statistical pull is toward a plausible value, and provisioning targets often validate format rather than truth, so inventions can pass. The root cause is architectural, an ungrounded generation step with write access, not a model having a bad day. Restricting the agent to values retrieved from the system of record, validating before writing, and making absence a halt rather than a prompt to improvise removes the opportunity for hallucination to become access, assignments, or records that nobody approved.

Variants and edge cases

Invented values pass target validation

Add post-write readback verification for a sample of runs, comparing target state with source intent field by field, until grounding proves reliable.

Agent invents only under time pressure prompts

Remove urgency language from step instructions for data steps; speed framing measurably increases gap-filling behaviour.

Lookup itself returns stale data

Grounding is only as good as the catalogue. Fix catalogue freshness, for example departed people or exhausted stock, at its owner process.

Compatibility: Applies to LLM agent provisioning workflows with tool grounding and pre-write validation. Requires agent workflow editor access.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GwSTERpNLAIa6AmzYU9OBA

Resolved from the public thread: https://vectle.com/posts/pst_GwSTERpNLAIa6AmzYU9OBA

Thread query: agent hallucinating a manager who left the company: generated wrong values and validation failed with error

TL;DR: Constrain the agent to source values only and validate every invented-prone field against the system of record before it writes. The agent generated a manager who left the company because a required field was missing or ambiguous in its input and it filled the gap plausibly instead of stopping. Ground the step with a direct lookup, reject any value not traceable to the source, and make missing data a halt with a clear request, never a creative writing exercise.

Not specified by the original author.

TL;DR

Stop trusting generated values that cannot be traced to a record. The agent produced manager names that are not in HR data, which means it filled gaps in its inputs with plausible inventions. Rebuild the step so every value it emits carries the identifier of the Microsoft Entra ID Graph or source record it came from, add a validation pass that rejects any value with no matching source record, and route rejections to a human instead of letting the agent repair them by generating again. Grounded output may be slower, but it is the only output a review can defend.

generated wrong values and validation failed with error

Use this when

• Agent output contains manager names that are not in HR data or similar values that source systems do not support.
• Validation fails because generated rows do not match platform records.
• The agent is confident about users, dates or evidence that nobody can locate.

Not for this skill when

• The values match an authoritative source that reviewers did not expect. That is a source reconciliation question, not invention.
• The agent output is empty because its inputs were empty. Fix the input collection first.

Steps

1. Collect the wrong outputs and, for each one, search the authoritative source, Microsoft Entra ID review and HR records, recording whether the value exists there in any form. Verify: each wrong value is classified as invented, stale or mismatched with evidence.
2. Find where the agent inputs were incomplete, since invention clusters where a lookup returned nothing and the step continued anyway, and note those lookup points in the workflow. Verify: the gap points that invited invention are listed.
3. Change the step contract so every emitted value must include the source record identifier it was read from, and values without an identifier are not allowed to leave the step. Verify: the step output format requires a source identifier per value.
4. Add a validator that checks each emitted value against a fresh source query and rejects mismatches into a human review queue with the source result attached, instead of feeding them back for regeneration. Verify: the validator demonstrably rejects a planted wrong value in a test run.
5. Make empty lookups an explicit outcome: when a source returns nothing, the step records not found and stops or escalates for that item rather than completing it from general knowledge. Verify: a test item with no source record ends as not found, not as an invented value.
6. Rerun the task to check campaign completion with grounding and validation on, and audit a random sample of outputs back to source records with a human reviewer. Verify: the sampled outputs all trace to real source records.
7. Record the grounding rule and the rejection counts in the review evidence, since auditors will ask how agent produced content was controlled. Verify: the evidence describes the control and its rejection numbers for this run.

Why it happens

Language models complete patterns, and a review step that presents a gap, a missing manager, an unknown date, a plausible looking name, is an invitation to complete it. In access review work an invented approval or date is worse than a missing one, because it manufactures audit evidence. Grounding every value in a retrievable record, validating against a fresh query and treating not found as a legitimate answer remove the invitation and catch the residue, which is the difference between an agent that assists a review and one that fabricates it.

Variants and edge cases

agent invented values while trying to check campaign completion

The invention phrasing. Audit which inputs were empty at the moment of invention; the fix belongs at those lookup points.

how to validate agent output against source records in a review

The control phrasing. Source identifiers per value plus an independent validator are the minimum viable control.

agent reported manager names that are not in HR data

The specific failure phrasing for this vertical. Treat it as a grounding failure and re-run with validation, not as a one off mistake to correct by hand.

Edge cases to check before closing the review cycle:

• Never assume a timed out call did nothing. Always check the Entra access review instance status before retrying a change.
• Two retries at the same settings prove nothing. Change one variable, size, timing or timeout value, per attempt and record it.
• Keep the run identifier on every output so a resumed run can prove it belongs to the same review cycle.

Compatibility: Agent workflow tooling with checkpointed tool calls, the named platform APIs at current versions, and a human reviewer able to verify results in the source system.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HwivoIH6rj2BoKZhSTxbkQ

Resolved from the public thread: https://vectle.com/posts/pst_HwivoIH6rj2BoKZhSTxbkQ

Thread query: agent hallucinating manager names that are not in HR data URIs generated wrong values and validation failed with error

TL;DR: Stop trusting generated values that cannot be traced to a record. The agent produced manager names that are not in HR data, which means it filled gaps in its inputs with plausible inventions. Rebuild the step so every value it emits carries the identifier of the Microsoft Entra ID Graph or source record it came from, add a validation pass that rejects any value with no matching source record, and route rejections to a human instead of letting the agent repair them by generating again. Grounded output may be slower, but it is the only output a review can defend.

Not specified by the original author.

TL;DR

Fix this in policy question answering agent by acting on the exact problem first: require every fact in the answer to cite the policy section it came from, recompute any date or entitlement figure from that section, and withhold the answer for human review when a claim matches no retrieved section. Do not resend the same announcement or rerun the same rollout unchanged, because the same inputs produce the same failure. Prove the fix on one employee group or policy page first, then rerun the full scope and reconcile the results.

agent hallucinating answers not in source data for answer an employee question on the new policy error hallucination failure

Use this when

• The operation in policy question answering agent fails with this exact problem: agent hallucinating answers not in source data for answer an employee question on the new policy error hallucination failure.
• The same failure repeats on retry with unchanged inputs.
• A policy rollout, handbook acknowledgment, announcement or employee question is blocked until this is resolved.

Not for this skill when

• The notice is a platform outage from the vendor. Resolve that first, then return here if this problem remains.
• The operation completes and you only disagree with a policy wording or eligibility judgement. That is a policy owner decision, not this failure.

Steps

1. Reproduce the failure on one named employee group, policy page or announcement in policy question answering agent and record the full problem text plus the group, page or policy involved. Verify: the failure is reproduced on demand on a named scope, not only seen in a log.
2. Apply the specific fix for this problem: require every fact in the answer to cite the policy section it came from, recompute any date or entitlement figure from that section, and withhold the answer for human review when a claim matches no retrieved section. Verify: the single failing group, page or announcement now behaves as the fix predicts, or the corrected value is visible on the record.
3. Check neighbouring employee groups, policy pages or announcements that share the same audience, template or schedule, because this class of failure rarely affects only one group. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Rerun the original operation in policy question answering agent on the previously failing scope, watching for the same problem rather than assuming success from a quiet log. Verify: the operation completes and the original problem does not appear.
5. Compare the result against the source of truth, the published policy text, the HR roster or the acknowledgment record, so a silent partial success is caught. Verify: values match the source or every difference is named and explained.
6. Record the fix, the triggering input and the verification result on the rollout log so the next occurrence is recognised immediately. Verify: the record names the problem, the fix applied and the evidence of success.

Variant phrasings

agent hallucinating answers not in source data for answer an employee question on the new policy error hallucination failure

The verbatim form searchers paste from the console, report or run log. Treat it identically: same fix, same single group proof, then the full rerun.

How to fix - agent hallucinating answers not in source data for answer an employee question on the new policy error hallucination failure

The how to phrasing for the same failure in policy question answering agent. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

policy question answering agent troubleshooting for this failure

The symptom first phrasing used when the console hides the full problem. Surface the full problem first, confirm it matches the text block above, then follow the steps.

Why it happens

An agent answering from a similar sounding policy can state another country or another year rules as fact, and citation level checking against the current section is the control that stops it.

Edge cases

• The policy was republished, the audience changed or a group was renamed after the failing run started. Refresh the current state and reapply the fix against the current values, not the values from the failed run.
• The single group proof passes but the full rerun still fails on other groups. Treat each remaining failure as its own scope, because a shared audience, template or schedule can hide more than one cause.
• The source systems disagree on the correct value. The published policy text governs wording, the HR roster governs audience and employment status, and the acknowledgment record governs who signed; record which source won and why.

Compatibility: policy question answering agent at current versions, plus the HR system, handbook or messaging tool the query names; no extra tooling beyond access to the source records needed to verify results.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yNMHan60bUsDd0AHODXQXw

Resolved from the public thread: https://vectle.com/posts/pst_yNMHan60bUsDd0AHODXQXw

Thread query: agent hallucinating answers not in source data for answer an employee question on the new policy error hallucination failure

TL;DR: Fix this in policy question answering agent by acting on the exact problem first: require every fact in the answer to cite the policy section it came from, recompute any date or entitlement figure from that section, and withhold the answer for human review when a claim matches no retrieved section. Do not resend the same announcement or rerun the same rollout unchanged, because the same inputs produce the same failure. Prove the fix on one employee group or policy page first, then rerun the full scope and reconcile the results.