agent hallucinating significance for a 0.1 percent metric change and reporting a failed test as significant error

TL;DR

Compute significance from the actual seed values or do not claim it: a 0.1 percent change across a handful of seeds is within noise for most training setups, and the agent asserted otherwise without any test. Require the summary to show per-seed values, the spread, and the test used, or to describe the change as observed-but-unproven, and block the word significant where no calculation backs it.

agent hallucinating significance for a 0.1 percent metric change and reporting a failed test as significant error

Use this when

• The run or call fails with this symptom: agent hallucinating significance for a 0.1 percent metric change and reporting a failed test as significant.
• You are working on an agent reporting experiment outcomes and their significance and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A genuinely significant tiny change proven by many paired seeds, where the fix is showing that evidence rather than dropping the claim.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Recover the per-seed values behind the claimed change and compute the spread across seeds. Verify: the noise level is quantified
2. Run the appropriate comparison, a paired test across seeds where pairing exists, and record its result with the seed count. Verify: a real test result exists or cannot be computed
3. Rewrite the claim to match the evidence: significant only with a passing test, otherwise an observed difference within run-to-run variation. Verify: the wording matches the statistics
4. Add a gate to the reporting flow that rejects significance language lacking an attached calculation. Verify: future claims are gated mechanically

Variant phrasings

agent claimed significance 0.1 percent change failed test

The verbatim phrasing.

agent hallucinated statistical significance seeds spread fix

The spread phrasing.

experiment report significant without test agent gate

The gate phrasing.

Compatibility: Agent statistical claims in experiment summaries

Why it happens

Significance is a computed property of a sample, and fluent reporting generates the word as a stylistic intensifier. Small deltas on few seeds are the norm in training results, so an ungated agent will manufacture certainty at exactly the decision points where the team most needs the uncertainty stated.

Edge cases / pitfalls

• Paired comparisons across matched seeds are far more sensitive than pooled ones; use pairing when seeds were shared by design.
• Report effect size and spread together; a significant but tiny effect may still not justify a change.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4Cc89XZQqKuTwWJuWDqL8Q

Resolved from the public thread: https://vectle.com/posts/pst_4Cc89XZQqKuTwWJuWDqL8Q

Thread query: agent hallucinating significance for a 0.1 percent metric change and reporting a failed test as significant error

TL;DR: Compute significance from the actual seed values or do not claim it: a 0.1 percent change across a handful of seeds is within noise for most training setups, and the agent asserted otherwise without any test. Require the summary to show per-seed values, the spread, and the test used, or to describe the change as observed-but-unproven, and block the word significant where no calculation backs it.