pipeline agent's dbt run timed out at step 47 of 60: resume from checkpoint

TL;DR

The agent's dbt run timed out at step 47 of 60, and rerunning all 60 steps wastes time and usually hits the same timeout. Resume only what failed: inspect the run artifacts for the failed models, then rerun with a selection limited to those models and their downstream dependents.

Symptom

An agent executing a dbt build times out partway through (step 47 of 60 here). The naive recovery reruns the whole build from step 1, burning the step budget again and timing out at roughly the same place. Completed models get rebuilt pointlessly while the actual failure never gets isolated.

When to use

• An agent-run dbt build times out partway through
• Reruns keep dying at roughly the same step
• The agent's per-step timeout is shorter than the full build

When not to use

• dbt fails at compile time before any model runs (thats a syntax or config error)
• The warehouse itself is down (no resume strategy helps)
• Every model fails (thats a systemic problem, investigate the warehouse first)

Steps

1. Find exactly which models failed instead of guessing. Read the run artifacts:

dbt ls --resource-type model --output name
cat target/run_results.json | python -c "import json,sys; d=json.load(sys.stdin); print([r['unique_id'] for r in d['results'] if r['status']=='error'])"

Expected output: the short list of failed model ids. Everything else succeeded and must not be rebuilt.

2. Resume with a selection limited to failures and their downstream dependents:

dbt build --select result:error+

Expected output: only the failed models and things that depend on them run. The + suffix pulls in downstream models so nothing runs on stale inputs.

3. Raise the agent's step timeout for the dbt step, or split the build into chunks the timeout can absorb:

dbt build --select tag:nightly_part1

Expected output: each chunk finishes inside the timeout. Chunking by tag or path beats one giant run when the agent harness enforces hard step limits.

4. Record the resume point on the agent's task state so a second timeout resumes from the new failure set, not from step 1:

dbt build --select result:error+ --target prod

Expected output: the second resume picks up where the first left off. Without persisted resume state, every retry starts over.

Variant phrasings

dbt run timed out at step N of M in CI

Same pattern. CI timeouts are usually fixed step budgets; chunk the run or raise the job timeout.

agent keeps rerunning successful dbt models

The agent lacks a resume strategy. Give it the result:error+ selection pattern as the default retry, not a full rerun.

Why it happens

Agents typically wrap shell commands with a fixed timeout and treat any timeout as total failure, so the retry path reruns the entire command. dbt builds are incremental by nature: successful models wrote their tables already, and only the failed ones need attention. The mismatch between the agent's all-or-nothing retry and dbt's incremental model is what turns one timeout into an endless loop.

Edge cases

• result:error only covers the last run in target/. If the agent wiped the target directory, there is no failure set to select from; rerun with --fail-fast off and let it rebuild the artifacts first.
• Snapshots and seeds in the failed set need the same selection treatment; they are not models but result:error+ still catches them.
• If the timeout was the warehouse queuing (not slow models), resuming immediately re-queues. Wait for warehouse capacity first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DtqW9INis37Qon1A0IRgfg