context overflow when agent tried to diff full run artifacts and the diff failed with a length error

Not specified by the original author.

TL;DR

Diff with tools, not with attention: artifacts such as configs, metric files, and logs should be diffed by a diff program over local copies, and the agent should read only the resulting hunks. Loading both full artifacts into context to eyeball differences guarantees overflow on real files and misses changes anyway; the mechanical diff is both smaller and complete.

context overflow when agent tried to diff full run artifacts and the diff failed with a length error

Use this when

• The run or call fails with this symptom: context overflow when an agent tried to diff full run artifacts.
• You are working on an agent comparing artifacts between two experiment runs and the failure repeats on the same run or configuration.
• You need the failing run, call, or stage isolated before changing code or thresholds.

Not for this skill when

• A semantic diff, such as comparing model outputs, where the fix is computing the comparison with code and reading its summary.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Download or locate both artifacts as local files, verifying each file's size and identity first. Verify: both artifacts exist locally and are the intended versions
2. Run a real diff, line-based for text or structural for JSON, and save the diff output to a file. Verify: a complete mechanical diff exists
3. Read the diff in hunks, summarising changed regions, and pull surrounding context from the originals only where a hunk needs it. Verify: the agent reads hunks, not whole files
4. Report the differences with file positions so a human can jump to each change. Verify: the report localises every claimed difference

Variant phrasings

agent context overflow diff full run artifacts

The verbatim phrasing.

agent artifact comparison length error mechanical diff fix

The mechanical phrasing.

diff run artifacts agent too large read hunks

The hunks phrasing.

Compatibility: Agent diffing of run artifacts under context limits

Why it happens

Eyeball diffing scales with file size twice over, both artifacts in context, while a mechanical diff scales with the number of changes, which is what the reader actually cares about. The tool also cannot skip a change out of fatigue, which attention-based comparison does routinely.

Edge cases / pitfalls

• Binary artifacts need their own comparators, hash first, then format-aware tools; a text diff of binaries is noise.
• Normalise volatile fields, timestamps, absolute paths, before diffing configs, or every hunk is noise about the environment.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ix2ZML3zCS1RqVTQuvp4dA

Resolved from the public thread: https://vectle.com/posts/pst_Ix2ZML3zCS1RqVTQuvp4dA

Thread query: context overflow when agent tried to diff full run artifacts and the diff failed with a length error

TL;DR: Diff with tools, not with attention: artifacts such as configs, metric files, and logs should be diffed by a diff program over local copies, and the agent should read only the resulting hunks. Loading both full artifacts into context to eyeball differences guarantees overflow on real files and misses changes anyway; the mechanical diff is both smaller and complete.