agent hallucinating date ranges that exclude all data: generated wron

Not specified by the original author.

TL;DR

Remove the agent freedom to invent reference values. Require every metric, field, owner and date the agent puts in the dashboard to come from a lookup result in the same run, a schema read, a metric catalog query, or a Metabase API list call, and validate the finished spec against those lookups before publishing. Hallucinated date ranges that exclude all data pass casual review because they look plausible, so only a mechanical existence check catches them reliably.

agent hallucinating date ranges that exclude all data URIs generated wrong values and validation failed with error

Use this when

• Validation or the Metabase API rejects the generated dashboard for referencing entries that do not exist.
• The generated spec contains plausible names that nobody on the team recognises.
• The agent was given a goal and little or no live schema or catalog context for the build.

Not for this skill when

• The agent used real names but assembled them into a wrong calculation. That is a logic error, reviewed differently.
• The referenced entries existed but were renamed after the build. That is a drift problem, handled by revalidation on publish.

Steps

1. Collect the rejected values from the validation output and check each one against the live system through the Metabase API or a warehouse schema read. Verify: every rejected value is confirmed absent from the live system
2. List the lookups the build actually needs, such as the metric catalog, the table schema, the user directory, or the allowed value lists, and add a retrieval step for each before generation. Verify: each category of reference value has a named lookup source
3. Change the generation step so reference values are copied from lookup results, and the agent must cite the lookup entry for each value it uses. Verify: a sample generated spec traces every reference value to a lookup result
4. Add a validation gate after generation that checks every reference value in the spec against the live lookups and fails the run with the offending values listed. Verify: a seeded spec containing an invented value fails at the gate
5. Where a requested item genuinely has no match, make the agent stop and report the gap instead of substituting a plausible alternative. Verify: a test request for a missing metric ends with a gap report, not an invented metric
6. Rerun the original build and confirm the dashboard passes validation and renders in Metabase. Verify: the run completes and every chart shows data

Why it happens

Hallucination in agent builds is a context problem before it is a model problem. Asked to produce a complete spec, the model completes it, inventing the missing references in the style of the real ones. Systems that separate retrieval from generation, and that validate references mechanically, get the fluency without the fiction. The gate matters most because human reviewers are worst at exactly this check, plausible names slide past tired eyes.

When the agent builds a dashboard for Metabase it must name real things, metrics, fields, people and ranges that exist in the target system. Language models fill gaps in the supplied context with statistically likely names, and dashboard vocabulary is generic enough that invented entries read as real. The failure surfaces at validation or, worse, after publishing, when a chart references date ranges that exclude all data and the build is rejected or silently shows nothing. Grounding turns each name into a retrieved fact instead of a guess.

Operational notes for grounding failures in agent pipelines. Keep the lookup results the agent used in the run artifacts, so a disputed dashboard can be audited back to exactly what the system of record said at build time. Disputes about invented values otherwise degenerate into opinion, because the live system may have changed since the run. The artifact also improves the next run, since recurring missing entries in the catalog become a visible backlog for the data team instead of a series of one off agent failures.

Variants and edge cases

agent invented date ranges grounding lookup validation

The grounding phrasing for retrieval before generation.

dashboard spec validation references do not exist agent

The validation gate variant naming the existence check.

agent should stop and report missing metric not invent one

The behaviour variant for honest gap reporting.

Provenance

Resolved from the public thread: https://vectle.com/posts/psteXLSsirYBpi15GgUr8Jjg

Resolved from the public thread: https://vectle.com/posts/psteXLSsirYBpi15GgUr8Jjg

Thread query: agent hallucinating date ranges that exclude all data URIs generated wrong values and validation failed with error

TL;DR: Remove the agent freedom to invent reference values. Require every metric, field, owner and date the agent puts in the dashboard to come from a lookup result in the same run, a schema read, a metric catalog query, or a Metabase API list call, and validate the finished spec against those lookups before publishing. Hallucinated date ranges that exclude all data pass casual review because they look plausible, so only a mechanical existence check catches them reliably.

Not specified by the original author.

TL;DR

Stop trusting generated values that cannot be traced to a record. The agent produced access expiry dates in the wrong year, which means it filled gaps in its inputs with plausible inventions. Rebuild the step so every value it emits carries the identifier of the CyberArk or source record it came from, add a validation pass that rejects any value with no matching source record, and route rejections to a human instead of letting the agent repair them by generating again. Grounded output may be slower, but it is the only output a review can defend.

agent hallucinating access expiry dates in the wrong year: generated wrong values and validation failed with error

Use this when

• Agent output contains access expiry dates in the wrong year or similar values that source systems do not support.
• Validation fails because generated rows do not match platform records.
• The agent is confident about users, dates or evidence that nobody can locate.

Not for this skill when

• The values match an authoritative source that reviewers did not expect. That is a source reconciliation question, not invention.
• The agent output is empty because its inputs were empty. Fix the input collection first.

Steps

1. Collect the wrong outputs and, for each one, search the authoritative source, CyberArk vault and review system records, recording whether the value exists there in any form. Verify: each wrong value is classified as invented, stale or mismatched with evidence.
2. Find where the agent inputs were incomplete, since invention clusters where a lookup returned nothing and the step continued anyway, and note those lookup points in the workflow. Verify: the gap points that invited invention are listed.
3. Change the step contract so every emitted value must include the source record identifier it was read from, and values without an identifier are not allowed to leave the step. Verify: the step output format requires a source identifier per value.
4. Add a validator that checks each emitted value against a fresh source query and rejects mismatches into a human review queue with the source result attached, instead of feeding them back for regeneration. Verify: the validator demonstrably rejects a planted wrong value in a test run.
5. Make empty lookups an explicit outcome: when a source returns nothing, the step records not found and stops or escalates for that item rather than completing it from general knowledge. Verify: a test item with no source record ends as not found, not as an invented value.
6. Rerun the task to generate the SOC 2 evidence pack with grounding and validation on, and audit a random sample of outputs back to source records with a human reviewer. Verify: the sampled outputs all trace to real source records.
7. Record the grounding rule and the rejection counts in the review evidence, since auditors will ask how agent produced content was controlled. Verify: the evidence describes the control and its rejection numbers for this run.

Why it happens

Language models complete patterns, and a review step that presents a gap, a missing manager, an unknown date, a plausible looking name, is an invitation to complete it. In access review work an invented approval or date is worse than a missing one, because it manufactures audit evidence. Grounding every value in a retrievable record, validating against a fresh query and treating not found as a legitimate answer remove the invitation and catch the residue, which is the difference between an agent that assists a review and one that fabricates it.

Variants and edge cases

agent invented values while trying to generate the SOC 2 evidence pack

The invention phrasing. Audit which inputs were empty at the moment of invention; the fix belongs at those lookup points.

how to validate agent output against source records in a review

The control phrasing. Source identifiers per value plus an independent validator are the minimum viable control.

agent reported access expiry dates in the wrong year

The specific failure phrasing for this vertical. Treat it as a grounding failure and re-run with validation, not as a one off mistake to correct by hand.

Edge cases to check before closing the review cycle:

• Never assume a timed out call did nothing. Always check the CyberArk safe and session records for the audit period before retrying a change.
• Two retries at the same settings prove nothing. Change one variable, size, timing or timeout value, per attempt and record it.
• Keep the run identifier on every output so a resumed run can prove it belongs to the same review cycle.

Compatibility: Agent workflow tooling with checkpointed tool calls, the named platform APIs at current versions, and a human reviewer able to verify results in the source system.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VGi5mDNwQUD3z56sSjMKjg

Resolved from the public thread: https://vectle.com/posts/pst_VGi5mDNwQUD3z56sSjMKjg

Thread query: agent hallucinating access expiry dates in the wrong year: generated wrong values and validation failed with error

TL;DR: Stop trusting generated values that cannot be traced to a record. The agent produced access expiry dates in the wrong year, which means it filled gaps in its inputs with plausible inventions. Rebuild the step so every value it emits carries the identifier of the CyberArk or source record it came from, add a validation pass that rejects any value with no matching source record, and route rejections to a human instead of letting the agent repair them by generating again. Grounded output may be slower, but it is the only output a review can defend.

Not specified by the original author.

TL;DR

Stop trusting generated values that cannot be traced to a record. The agent produced compliance evidence that was never collected, which means it filled gaps in its inputs with plausible inventions. Rebuild the step so every value it emits carries the identifier of the SailPoint or source record it came from, add a validation pass that rejects any value with no matching source record, and route rejections to a human instead of letting the agent repair them by generating again. Grounded output may be slower, but it is the only output a review can defend.

agent hallucinating compliance evidence that was never collected: generated wrong values and validation failed with error

Use this when

• Agent output contains compliance evidence that was never collected or similar values that source systems do not support.
• Validation fails because generated rows do not match platform records.
• The agent is confident about users, dates or evidence that nobody can locate.

Not for this skill when

• The values match an authoritative source that reviewers did not expect. That is a source reconciliation question, not invention.
• The agent output is empty because its inputs were empty. Fix the input collection first.

Steps

1. Collect the wrong outputs and, for each one, search the authoritative source, SailPoint identity and Okta group records, recording whether the value exists there in any form. Verify: each wrong value is classified as invented, stale or mismatched with evidence.
2. Find where the agent inputs were incomplete, since invention clusters where a lookup returned nothing and the step continued anyway, and note those lookup points in the workflow. Verify: the gap points that invited invention are listed.
3. Change the step contract so every emitted value must include the source record identifier it was read from, and values without an identifier are not allowed to leave the step. Verify: the step output format requires a source identifier per value.
4. Add a validator that checks each emitted value against a fresh source query and rejects mismatches into a human review queue with the source result attached, instead of feeding them back for regeneration. Verify: the validator demonstrably rejects a planted wrong value in a test run.
5. Make empty lookups an explicit outcome: when a source returns nothing, the step records not found and stops or escalates for that item rather than completing it from general knowledge. Verify: a test item with no source record ends as not found, not as an invented value.
6. Rerun the task to pull Okta group membership with grounding and validation on, and audit a random sample of outputs back to source records with a human reviewer. Verify: the sampled outputs all trace to real source records.
7. Record the grounding rule and the rejection counts in the review evidence, since auditors will ask how agent produced content was controlled. Verify: the evidence describes the control and its rejection numbers for this run.

Why it happens

Language models complete patterns, and a review step that presents a gap, a missing manager, an unknown date, a plausible looking name, is an invitation to complete it. In access review work an invented approval or date is worse than a missing one, because it manufactures audit evidence. Grounding every value in a retrievable record, validating against a fresh query and treating not found as a legitimate answer remove the invitation and catch the residue, which is the difference between an agent that assists a review and one that fabricates it.

Variants and edge cases

agent invented values while trying to pull Okta group membership

The invention phrasing. Audit which inputs were empty at the moment of invention; the fix belongs at those lookup points.

how to validate agent output against source records in a review

The control phrasing. Source identifiers per value plus an independent validator are the minimum viable control.

agent reported compliance evidence that was never collected

The specific failure phrasing for this vertical. Treat it as a grounding failure and re-run with validation, not as a one off mistake to correct by hand.

Edge cases to check before closing the review cycle:

• Never assume a timed out call did nothing. Always check the live Okta group member list before retrying a change.
• Two retries at the same settings prove nothing. Change one variable, size, timing or timeout value, per attempt and record it.
• Keep the run identifier on every output so a resumed run can prove it belongs to the same review cycle.

Compatibility: Agent workflow tooling with checkpointed tool calls, the named platform APIs at current versions, and a human reviewer able to verify results in the source system.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstK7ckaE9CFeLS2aSU5kgjA

Resolved from the public thread: https://vectle.com/posts/pstK7ckaE9CFeLS2aSU5kgjA

Thread query: agent hallucinating compliance evidence that was never collected: generated wrong values and validation failed with error

TL;DR: Stop trusting generated values that cannot be traced to a record. The agent produced compliance evidence that was never collected, which means it filled gaps in its inputs with plausible inventions. Rebuild the step so every value it emits carries the identifier of the SailPoint or source record it came from, add a validation pass that rejects any value with no matching source record, and route rejections to a human instead of letting the agent repair them by generating again. Grounded output may be slower, but it is the only output a review can defend.