agent failed to notice metric direction error and flagged lower loss as a regression failure

TL;DR

Carry direction with the metric everywhere: lower-is-better and higher-is-better must be metadata the comparison reads, not knowledge the agent improvises per run. Build a metric direction table for the project, loss and error down, accuracy and score up, and make the regression check consult it, flagging any metric absent from the table for a human decision instead of guessing.

agent failed to notice metric direction error and flagged lower loss as a regression failure

Use this when

• The run or call fails with this symptom: agent failed to notice a metric direction error and flagged lower loss as a regression.
• You are working on an agent judging regressions across experiment metrics and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A metric whose direction genuinely flips by context, such as a calibration error target, where the table needs a per-context entry.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. List the metrics the agent judges and write down the correct direction for each from the project's conventions. Verify: a direction table exists
2. Find where the agent's check assumed a direction and compare it with the table for the loss family. Verify: the inverted assumption is located
3. Wire the check to the table and make unknown metrics stop the check with a question rather than a guess. Verify: directions are data-driven
4. Replay the mis-flagged run through the corrected check and confirm it now passes, and a worsened-loss run flags. Verify: both directions are verified against reality

Variant phrasings

agent flagged lower loss as regression metric direction error

The verbatim phrasing.

agent regression check wrong direction loss lower better fix

The direction-table phrasing.

experiment agent direction metadata metrics regression false

The metadata phrasing.

Compatibility: Agent metric interpretation with per-metric direction metadata

Why it happens

Direction is a property of the metric, not of the comparison code, but agents routinely hardcode the accuracy intuition, more is better, and apply it to losses. The resulting false regressions are confident and systematic, firing on every improvement until the alerts are ignored.

Edge cases / pitfalls

• Compound metrics, such as a weighted score containing a loss term, need their direction derived and recorded at creation.
• When a run renames a metric, the direction lookup must follow the rename or the check silently loses the metric.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_7TuMY6xt7tIFCuXfZzUFZw

Resolved from the public thread: https://vectle.com/posts/pst_7TuMY6xt7tIFCuXfZzUFZw

Thread query: agent failed to notice metric direction error and flagged lower loss as a regression failure

TL;DR: Carry direction with the metric everywhere: lower-is-better and higher-is-better must be metadata the comparison reads, not knowledge the agent improvises per run. Build a metric direction table for the project, loss and error down, accuracy and score up, and make the regression check consult it, flagging any metric absent from the table for a human decision instead of guessing.