when agent tried to compare baseline and candidate runs it failed with a missing metric error
Fetch both runs' metric inventories before comparing and fail loudly on absence: the comparison died, or worse, silently compared substitutes, because the candidate never logged the metric the baseline had. Use when an agent comparing a baseline and a candidate run for a promotion decision. Not for a metric renamed between the runs' framework versions, where the fix is a name mapping rather than a re-evaluation.
TL;DR
Fetch both runs' metric inventories before comparing and fail loudly on absence: the comparison died, or worse, silently compared substitutes, because the candidate never logged the metric the baseline had. Align on metrics present in both, state which baseline metrics have no counterpart, and if the missing metric is the decision metric, send the candidate back for evaluation instead of comparing around it.
when agent tried to compare baseline and candidate runs it failed with a missing metric errorUse this when
- The run or call fails with this symptom: agent tried to compare baseline and candidate runs and failed with a missing metric error.
- You are working on an agent comparing a baseline and a candidate run for a promotion decision and the failure repeats on the same run or configuration.
- You need the failing run, call, or stage isolated before changing code or thresholds.
Not for this skill when
- A metric renamed between the runs' framework versions, where the fix is a name mapping rather than a re-evaluation.
- The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.
Steps
- List the metric keys recorded on each run and build the intersection and the two differences. Verify: metric coverage per run is explicit
- Decide per missing metric: map a rename, accept the gap with a note, or block the comparison if it is the decision metric. Verify: each gap has a stated handling
- Run the comparison over the aligned set only, with the mapping and gaps recorded in the output. Verify: the comparison completes honestly
- If blocked, trigger or request the missing evaluation on the candidate and rerun once it lands. Verify: the decision metric exists on both sides
Variant phrasings
agent compare baseline candidate missing metric error
The verbatim phrasing.
run comparison failed metric not on candidate agent fix
The coverage phrasing.
baseline candidate comparison metric alignment agent
The alignment phrasing.
Compatibility: Agent run comparison requiring matched metrics on both sides
Why it happens
Two runs log what their code versions logged, so metric sets drift apart over time exactly like schemas. A comparison that assumes symmetry breaks on the drift, and one that silently substitutes a near metric makes a confident, wrong decision instead.
Edge cases / pitfalls
- Evaluation metrics are the usual absentees because they are logged by a separate job that may not have run.
- Direction matters per metric; an aligned set still misleads if one metric's better direction is inverted.
Provenance
Resolved from the public thread: https://vectle.com/posts/pstlwxEwsXWcoodUkbnsiNKQ