agent hallucinating dataset version for run logged before versioning enabled error

Not specified by the original author.

TL;DR

Say unknown when versioning was off: the agent invented a dataset version for a run logged before dataset versioning existed, so mark the dataset version as unrecorded for those runs and bound any comparison to what the run actually stored. A missing version is a fact about the run. Filling it with the current version rewrites history and poisons every comparison that trusts the field.

agent hallucinating dataset version for run logged before versioning enabled error

Use this when

• The check or run fails with this symptom: an agent hallucinates a dataset version for a run logged before versioning was enabled.
• You are working on an agent workflow reporting or comparing runs that span a dataset versioning rollout and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• Runs logged after versioning was enabled whose version is stored in a field the agent did not check, which needs the field found, not an unknown label.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Check the run record for a dataset version field and the date versioning was enabled. Verify: the run predates versioning and carries no version
2. List every report where the agent stated a version for such a run. Verify: each invented value is identified
3. Correct those records to unrecorded, with the run date as the only temporal fact. Verify: reports no longer claim a version
4. Where the dataset can be inferred from immutable evidence, such as a stored data hash, record it as inferred with that evidence. Verify: inferred values are labelled and sourced
5. Add a workflow rule that missing fields are reported as missing. Verify: the agent output uses unrecorded instead of a guess

Variant phrasings

agent hallucinating dataset version run before versioning

The shorter phrasing focused on the named failure.

invented dataset version agent report error

The symptom phrasing a searcher tries next.

dataset version unrecorded older runs fix

The variant naming the failing stage or tool directly.

Compatibility: Dataset versioning in experiment trackers, agent reporting workflows

Why it happens

Versioning rollouts split history into runs that know their data version and runs that cannot. Agents asked to complete a comparison table experience the empty field as a gap to fill, and the most available value, the current dataset version, is almost certainly wrong for the oldest runs.

Edge cases / pitfalls

• A data hash or a snapshot id stored by older tooling can identify a dataset honestly; look for evidence before settling for unrecorded.
• Comparisons spanning the rollout should state the boundary date so readers know which runs are versioned.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstUABjthcKz-udoutllrIGA

Resolved from the public thread: https://vectle.com/posts/pstUABjthcKz-udoutllrIGA

Thread query: agent hallucinating dataset version for run logged before versioning enabled error

TL;DR: Say unknown when versioning was off: the agent invented a dataset version for a run logged before dataset versioning existed, so mark the dataset version as unrecorded for those runs and bound any comparison to what the run actually stored. A missing version is a fact about the run. Filling it with the current version rewrites history and poisons every comparison that trusts the field.