agent failed to group runs by dataset version because grouping key error split identical runs

TL;DR

Normalise the grouping key before grouping: identical runs split because their dataset version is recorded in variant forms, a version string in one run, a path or hash in another, or a tag on some and a param on others. Define one canonical key derived from all the places the version is recorded, with a stated precedence, and group by that, reporting runs whose version cannot be determined instead of silently splitting them.

agent failed to group runs by dataset version because grouping key error split identical runs

Use this when

• The run or call fails with this symptom: agent failed to group runs by dataset version because a grouping key error split identical runs.
• You are working on an agent grouping runs by dataset version for comparison and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• Runs that genuinely used different dataset snapshots under one version label, where the fix is recording content hashes, not key normalisation.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Collect every way the dataset version appears across the runs: params, tags, and config fields. Verify: all recording locations are inventoried
2. Define the canonical key and the precedence among sources, plus the rule for runs missing all of them. Verify: one key derivation is specified
3. Regroup with the canonical key and compare group counts and memberships with the previous grouping. Verify: identical runs land together and differences are explained
4. Flag undeterminable runs in the output rather than assigning them a guessed group. Verify: unknown versions are visible, not hidden

Variant phrasings

agent group runs dataset version grouping key split identical

The verbatim phrasing.

runs split groups dataset version recorded differently agent

The recording-drift phrasing.

agent grouping error dataset version canonical key fix

The canonical-key phrasing.

Compatibility: Agent grouping of experiment runs by dataset version metadata

Why it happens

Grouping is exact matching over whatever string the key happens to be, so recording drift, the same fact written three ways across code versions, shatters groups mechanically. The data was consistent; the key was not, and no amount of downstream care repairs a split introduced at grouping time.

Edge cases / pitfalls

• Paths containing the version and versions containing paths both occur; derive the key by parsing, not by raw string equality.
• When the version is a hash in some runs, a mapping table from hash to version may be needed; build it from the dataset registry.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_QeY7pXHrZR2LC0MiZSwWxA