fix-proposal agent crashed with error parsing a 2gb dbt artifacts log
Teaches a fix-proposal agent to handle giant dbt log files by streaming with grep instead of loading them, and by reading structured run_results.json instead of raw logs. Use when an agent crashes parsing multi-gigabyte dbt logs. Not for small logs, which can be read directly.
TL;DR
Never load a 2GB log into context. The agent should extract the failing section with grep (error lines plus surrounding context), or better, read the structured run_results.json which already has per-node status, timing, and compiled SQL references.
Error
fix-proposal agent crashed with error parsing a 2gb dbt artifacts logSteps
- Stop trying to read the whole file. First check its size and confirm the crash cause. Expected: you confirm it is a size problem, not a format problem.
- Extract failures with grep:
grep -n -i "error\|fail" dbt.log | head -50for an overview. Expected: a short list of failure locations. - Pull context around the interesting lines:
sed -n '[START],[END]p' dbt.log. Expected: the full error for the failing node. - Prefer the structured artifacts:
target/run_results.jsonhas per-node results with status and message fields. Expected: machine-readable failure data without log parsing. - Write the fix proposal from the structured results plus the extracted log window. Expected: a grounded proposal with no giant reads.
When to use
- An agent crashes or times out on huge dbt log or artifact files.
- Logs are gigabytes because of debug output or long runs.
When not to use
- Logs are small (read them directly).
- The needed data is already in run_results.json (skip the logs entirely).
Tool compatibility
- dbt Core 1.0 and later (run_results.json schema is versioned), standard Unix tools.
Variant phrasings
Agent OOM on target/ directory reads
Same fix: read the structured JSON summaries, not the raw files.
Parsing dbt.log with a strict parser
Logs are semi-structured; grep plus structured artifacts beats strict parsing.
Why it happens
dbt debug logs every query and render, so long runs produce enormous logs. Agents that treat the log as a document to read whole blow their context on megabytes of repetitive lines.
Edge cases
- Log rotation may split the failure across files; check timestamps to find the right one.
run_results.jsononly covers the last run; rerun the failing selection if the artifacts are stale.- Binary or truncated lines in logs break naive parsers; grep tolerates them.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_w0UwkQmDdZMUcfBiDJsmEA