agent failed to parse dbt cloud run logs: truncated json lines

TL;DR

dbt Cloud run logs can contain truncated JSON lines (cut off mid-record). The agent should parse line-delimited JSON defensively: try each line, skip and count the malformed ones, and get authoritative data from the run artifacts API instead of depending on the logs.

Error

agent failed to parse dbt cloud run logs: truncated json lines

Steps

1. Read the logs as plain text lines, not as one JSON document. Expected: line-level handling from the start.
2. Parse each line independently with error handling; on failure, record the line number and continue. Expected: good lines parse, bad lines are skipped with a count.
3. If the truncated lines carry critical data, fetch the structured run artifacts for the run instead. Expected: complete, well-formed data from the API.
4. Report the skip count alongside results. Expected: the agent is honest about what it could not parse.
5. Continue the analysis on the clean data. Expected: no crash, no silent data loss.

When to use

• Parsing dbt Cloud run logs fails on truncated or malformed JSON lines.
• Any line-delimited JSON stream with occasional corruption.

When not to use

• Logs are well-formed (parse them normally).
• The needed data is only in artifacts (skip the logs entirely).

Tool compatibility

• dbt Cloud API; any JSON parsing library with per-line error handling.

Variant phrasings

Log stream cut off mid-run

Treat the cutoff as the end of available data; do not invent the rest.

Interleaved log lines from parallel steps

Parse per line and tag by step; ordering is not guaranteed.

Why it happens

Log shipping truncates long lines and can cut streams mid-write. Strict parsers treat one bad line as a fatal error, but the rest of the log is usually fine.

Edge cases

• A high skip count means the logs are unreliable for the question; switch to artifacts.
• Keep a sample of skipped lines for debugging the parser later.
• Never silently drop lines that might contain the failure; count and report them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_n5-ABCXzR83aIrpne8TeSw