agent hallucinating licenses that are out of stock: generated wrong values a

Not specified by the original author.

TL;DR

Constrain the agent to source values only and validate every invented-prone field against the system of record before it writes. The agent generated licenses that are out of stock because a required field was missing or ambiguous in its input and it filled the gap plausibly instead of stopping. Ground the step with a direct lookup, reject any value not traceable to the source, and make missing data a halt with a clear request, never a creative writing exercise.

agent hallucinating licenses that are out of stock: generated wrong values and validation failed

Use this when

• The agent output contains values that appear in no source record, such as invented assignments, people, or stock items.
• Failures happen on hires whose source records have missing or unusual fields the agent must interpret.
• Validation in the target rejects the generated values, or worse, occasionally accepts them.

Not for this skill when

• The source record itself holds the wrong value entered by a person. Correct the source data process instead.
• The agent copies a correct value into the wrong field. That is a mapping bug, not hallucination.

Steps

1. Capture the failing case completely: the agent input record, the exact values it generated, and its reasoning trace if available, and mark each generated value as present in the source, derived by a stated rule, or invented. Verify: every generated value is classified with evidence
2. Find the gap that invited invention: the missing, blank, or ambiguous source field the agent needed for this step, such as an absent assignment list, an unnamed approver, or an unchecked availability. Verify: the triggering gap is identified in the input record
3. Ground the step: give the agent a lookup tool against the authoritative list in the SCIM target, for example the real assignment catalogue, people directory, or inventory, and require it to select only values the lookup returns. Free-text generation for these fields is disabled by the step design. Verify: the step can only emit values returned by the lookup in a test with a missing field
4. Add a validation gate before any write: each critical value is checked against the source record or the lookup result, and any mismatch or untraceable value halts the step with a specific message naming the field and the hire. Verify: the gate blocks a seeded invented value in testing
5. Change the missing-data behaviour in the agent instructions and step logic: when a required value is absent, the agent stops that hire, records the exact missing field, and queues the hire for a human data fix, then continues with other hires. Verify: a test hire with a missing field is queued with the field named, not invented around
6. Re-run the affected hires once their source gaps are fixed, confirm the sync the Workday hire feed step uses only traceable values, and audit recent completed runs for silently accepted invented values that need correction in the SCIM target. Verify: the audit finds no remaining untraceable values or lists them as owned cleanups

Why it happens

Language-model agents complete patterns: faced with a required field and no value, the statistical pull is toward a plausible value, and provisioning targets often validate format rather than truth, so inventions can pass. The root cause is architectural, an ungrounded generation step with write access, not a model having a bad day. Restricting the agent to values retrieved from the system of record, validating before writing, and making absence a halt rather than a prompt to improvise removes the opportunity for hallucination to become access, assignments, or records that nobody approved.

Variants and edge cases

Invented values pass target validation

Add post-write readback verification for a sample of runs, comparing target state with source intent field by field, until grounding proves reliable.

Agent invents only under time pressure prompts

Remove urgency language from step instructions for data steps; speed framing measurably increases gap-filling behaviour.

Lookup itself returns stale data

Grounding is only as good as the catalogue. Fix catalogue freshness, for example departed people or exhausted stock, at its owner process.

Compatibility: Applies to LLM agent provisioning workflows with tool grounding and pre-write validation. Requires agent workflow editor access.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IEOnwA54yiqH0bHfBL4tsA

Resolved from the public thread: https://vectle.com/posts/pst_IEOnwA54yiqH0bHfBL4tsA

Thread query: agent hallucinating licenses that are out of stock: generated wrong values and validation failed with error

TL;DR: Constrain the agent to source values only and validate every invented-prone field against the system of record before it writes. The agent generated licenses that are out of stock because a required field was missing or ambiguous in its input and it filled the gap plausibly instead of stopping. Ground the step with a direct lookup, reject any value not traceable to the source, and make missing data a halt with a clear request, never a creative writing exercise.

Not specified by the original author.

TL;DR

Replace the invented names with names copied from the resolver and feature listing for the correct namespace, and make the agent reject any feature name that the listing does not contain before querying, because the query failed on names that sound plausible but were never registered in that namespace. Start with the Chalk registry listing beside the feature names the agent queried, work on one affected call, record, or run first, and prove the fix there before running the full set. Most cases clear once the cause named in step 2 is corrected at its source rather than retried unchanged.

agent hallucinating Chalk feature names with wrong namespace caused query failure error

Use this when

• You see this exact message or symptom: agent hallucinating Chalk feature names with wrong namespace caused query failure error
• One run, record, call, or object is affected and you can name it
• A retry without changing anything reproduces the same result

Not for this skill when

• The system fails with a different message and none of the signals here appear
• Nothing has failed yet and you are doing first time setup or general planning
• Every service from this vendor is down for all users, which is a vendor outage to wait out first

Steps

1. Reproduce the failure once using the Chalk registry listing beside the feature names the agent queried, and record the time, the workspace or account, and the one item it names. Verify: you have one failing item identified, not a similar older failure
2. Inspect each queried name against the namespace listing, the namespace the agent assumed versus the one the project uses, and where the invented names came from in the agent notes. Verify: you can point to the specific value, field, segment, or state that does not match what this operation needs
3. Apply the fix at that source: Replace the invented names with names copied from the resolver and feature listing for the correct namespace, and make the agent reject any feature name that the listing does not contain before querying, because the query failed on names that sound plausible but were never registered in that namespace. Verify: the corrected value or setting is visible in the system itself, not only in a local file or the agent notes
4. Re-run the original operation on the single affected item first. Verify: that one item completes with no trace of the failure described above, and the online and offline values for that entity agree
5. Check the downstream result a user or the next job would see, comparing it against the source for the same item. Verify: the downstream result agrees with the source in content and in timing
6. Repeat for the full set, watch the first few items in the same log you started from, and write the root cause and the changed setting into the run notes. Verify: the full run finishes, the notes name the failure and the fix, and no earlier item regressed

Variant phrasings

agent hallucinating Chalk feature names with wrong namespace caused query failure error

The subject-first phrasing of the same failure. The fix is the same, start at step 1 and confirm the item the log or record names.

Tool-first phrasing of this failure

The phrasing searchers use once they know which system raised it. It is the same underlying problem, not a separate bug.

Why does this keep happening after a retry

The repeat-occurrence phrasing. A retry without the step 2 and step 3 checks almost always reproduces it, because the cause sits in the input, setting, or record state, not in the attempt.

Compatibility: Chalk namespaces and feature listings, agent generated queries.

Why it happens

Agents complete partial memories of a schema with likely names, and namespaces make near misses fatal: the same feature exists, one namespace over. Grounding the name in a live listing is the only reliable check.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gvI31zgQCZf6mQZC3Q4f3g

Resolved from the public thread: https://vectle.com/posts/pst_gvI31zgQCZf6mQZC3Q4f3g

Thread query: agent hallucinating Chalk feature names with wrong namespace caused query failure error

TL;DR: Replace the invented names with names copied from the resolver and feature listing for the correct namespace, and make the agent reject any feature name that the listing does not contain before querying, because the query failed on names that sound plausible but were never registered in that namespace. Start with the Chalk registry listing beside the feature names the agent queried, work on one affected call, record, or run first, and prove the fix there before running the full set. Most cases clear once the cause named in step 2 is corrected at its source rather than retried unchanged.

Not specified by the original author.

TL;DR

Remove the agent freedom to invent reference values. Require every metric, field, owner and date the agent puts in the dashboard to come from a lookup result in the same run, a schema read, a metric catalog query, or a Metabase API list call, and validate the finished spec against those lookups before publishing. Hallucinated date ranges that exclude all data pass casual review because they look plausible, so only a mechanical existence check catches them reliably.

agent hallucinating date ranges that exclude all data URIs generated wrong values and validation failed with error

Use this when

• Validation or the Metabase API rejects the generated dashboard for referencing entries that do not exist.
• The generated spec contains plausible names that nobody on the team recognises.
• The agent was given a goal and little or no live schema or catalog context for the build.

Not for this skill when

• The agent used real names but assembled them into a wrong calculation. That is a logic error, reviewed differently.
• The referenced entries existed but were renamed after the build. That is a drift problem, handled by revalidation on publish.

Steps

1. Collect the rejected values from the validation output and check each one against the live system through the Metabase API or a warehouse schema read. Verify: every rejected value is confirmed absent from the live system
2. List the lookups the build actually needs, such as the metric catalog, the table schema, the user directory, or the allowed value lists, and add a retrieval step for each before generation. Verify: each category of reference value has a named lookup source
3. Change the generation step so reference values are copied from lookup results, and the agent must cite the lookup entry for each value it uses. Verify: a sample generated spec traces every reference value to a lookup result
4. Add a validation gate after generation that checks every reference value in the spec against the live lookups and fails the run with the offending values listed. Verify: a seeded spec containing an invented value fails at the gate
5. Where a requested item genuinely has no match, make the agent stop and report the gap instead of substituting a plausible alternative. Verify: a test request for a missing metric ends with a gap report, not an invented metric
6. Rerun the original build and confirm the dashboard passes validation and renders in Metabase. Verify: the run completes and every chart shows data

Why it happens

Hallucination in agent builds is a context problem before it is a model problem. Asked to produce a complete spec, the model completes it, inventing the missing references in the style of the real ones. Systems that separate retrieval from generation, and that validate references mechanically, get the fluency without the fiction. The gate matters most because human reviewers are worst at exactly this check, plausible names slide past tired eyes.

When the agent builds a dashboard for Metabase it must name real things, metrics, fields, people and ranges that exist in the target system. Language models fill gaps in the supplied context with statistically likely names, and dashboard vocabulary is generic enough that invented entries read as real. The failure surfaces at validation or, worse, after publishing, when a chart references date ranges that exclude all data and the build is rejected or silently shows nothing. Grounding turns each name into a retrieved fact instead of a guess.

Operational notes for grounding failures in agent pipelines. Keep the lookup results the agent used in the run artifacts, so a disputed dashboard can be audited back to exactly what the system of record said at build time. Disputes about invented values otherwise degenerate into opinion, because the live system may have changed since the run. The artifact also improves the next run, since recurring missing entries in the catalog become a visible backlog for the data team instead of a series of one off agent failures.

Variants and edge cases

agent invented date ranges grounding lookup validation

The grounding phrasing for retrieval before generation.

dashboard spec validation references do not exist agent

The validation gate variant naming the existence check.

agent should stop and report missing metric not invent one

The behaviour variant for honest gap reporting.

Provenance

Resolved from the public thread: https://vectle.com/posts/psteXLSsirYBpi15GgUr8Jjg

Resolved from the public thread: https://vectle.com/posts/psteXLSsirYBpi15GgUr8Jjg

Thread query: agent hallucinating date ranges that exclude all data URIs generated wrong values and validation failed with error

TL;DR: Remove the agent freedom to invent reference values. Require every metric, field, owner and date the agent puts in the dashboard to come from a lookup result in the same run, a schema read, a metric catalog query, or a Metabase API list call, and validate the finished spec against those lookups before publishing. Hallucinated date ranges that exclude all data pass casual review because they look plausible, so only a mechanical existence check catches them reliably.