agent hallucinating sweep configuration values never set caused comparison failure error

TL;DR

Cite configuration only from the run record: the agent invented sweep values that were never set, so read every configuration value it reports from the stored sweep or run configuration, and mark unset values as defaults with their source. A plausible hyperparameter is still a fabrication when the run never set it. Sweep comparisons built on invented values misattribute every result they explain.

agent hallucinating sweep configuration values never set caused comparison failure error

Use this when

• The check or run fails with this symptom: an agent hallucinates sweep configuration values never set and causes a comparison failure.
• You are working on an agent workflow comparing or reporting sweep configurations and results and the failure repeats on the same data or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A value set by a framework default that the run record omits, which should be reported as the framework default, sourced from the framework documentation for that version.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. List the configuration values the agent reported for one run. Verify: each value is queued for verification
2. Read the same values from the stored run configuration. Verify: invented or mismatched values are identified
3. Correct the comparison with the recorded values, and label defaults as defaults. Verify: the comparison reflects the real sweep
4. Change the workflow so configuration fields are copied from run records by code, not written by the agent. Verify: generated reports carry record sourced values
5. Spot check a second run end to end. Verify: every reported value traces to its run record

Variant phrasings

agent hallucinating sweep configuration values never set

The shorter phrasing focused on the named failure.

invented hyperparameter comparison failure agent

The symptom phrasing a searcher tries next.

sweep values from run record fix

The variant naming the failing stage or tool directly.

Compatibility: Sweep configuration records in W&B and similar tools, agent reporting workflows

Why it happens

Sweep spaces are wide and run records are tedious to read, so an agent filling a comparison table from memory of the sweep design writes the values that should have been run. Where the sweep was edited mid flight or a run overrode a value, the report describes a sweep that never happened.

Edge cases / pitfalls

• The sweep design and the run record differ exactly when something interesting happened; the record wins every time.
• Defaults are values too; report them as defaults so readers do not credit the sweep with exploring them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstBmnhV7eKCElZmnbjql1-A

Resolved from the public thread: https://vectle.com/posts/pstBmnhV7eKCElZmnbjql1-A

Thread query: agent hallucinating sweep configuration values never set caused comparison failure error

TL;DR: Cite configuration only from the run record: the agent invented sweep values that were never set, so read every configuration value it reports from the stored sweep or run configuration, and mark unset values as defaults with their source. A plausible hyperparameter is still a fabrication when the run never set it. Sweep comparisons built on invented values misattribute every result they explain.