agent's diff parser broke with error on the dbt cloud job log format
Teaches an agent to stop strictly parsing dbt Cloud job logs and instead read structured run artifacts, treating logs as tolerant text. Use when an agent's log parser breaks on log format changes. Not for genuinely malformed API responses.
TL;DR
Human-readable logs are not a stable API; their format changes. The agent should get structured data from the run artifacts API (run_results.json equivalents) and treat log text as best-effort: tolerant patterns, skip-what-you-cannot-parse, never crash.
Error
agent's diff parser broke with error on the dbt cloud job log formatSteps
- Switch the primary data source from raw logs to the structured artifacts endpoint for the run. Expected: stable, versioned fields instead of prose.
- Where logs are still needed, parse with tolerant patterns (search for known markers, ignore unknown lines). Expected: format drift degrades detail, not the whole parse.
- Pin to documented fields only; never depend on the exact wording of a log line. Expected: the parser survives wording changes.
- Wrap parsing in error handling that logs the unparsable section and continues. Expected: one weird line never kills the run.
- Re-run the agent's analysis against the artifacts. Expected: it completes on structured data.
When to use
- An agent parser breaks when dbt Cloud changes log formatting.
- Any automation that scrapes human-readable logs.
When not to use
- The API itself returns malformed JSON (a real API bug; report it).
- The agent needs live streaming output (then tolerate the format explicitly).
Tool compatibility
- dbt Cloud API artifacts endpoints; any parsing language.
Variant phrasings
Log format changed between dbt versions
Version-pin the parser's expectations, or better, stop parsing logs.
Regex parser fails on a new log line
Tolerant parsing (find markers, skip the rest) is the durable fix.
Why it happens
Log lines are written for humans and change without notice. Strict parsers encode today's format as law, so every upstream wording tweak becomes an agent crash.
Edge cases
- Artifacts may lag the live run by a bit; poll for completion before fetching them.
- Some detail only exists in logs (adapter debug lines); extract it with loose searches, not strict grammars.
- Keep a sample of the new format when the parser degrades, so a human can update the markers.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_9F8mZlsaekIGOKtKUjGNMw