agent's dbt compile timed out on a 500-model project: no partial manifest saved

Not specified by the original author.

TL;DR

A full-project compile on a large project exceeds the agent timeout, and because manifest.json is only written at the end, a killed compile leaves nothing. Compile in slices with node selection, raise the step timeout, and enable partial parsing so repeat compiles reuse the saved manifest. Structure the agent loop around per-model compiles instead of one giant one.

agent's dbt compile timed out on a 500-model project: no partial manifest saved

Use this when

• an agent or automation times out running dbt compile on a project with hundreds of models
• after the timeout, target/manifest.json is missing or stale
• the agent needs compiled SQL for only a few models but compiles everything

Not for this skill when

• a single model compile hangs on a macro loop, debug that macro instead
• dbt itself crashes with a Python traceback, that is a bug or version issue, not a timeout budget issue
• you control the runner and can simply give the one compile more time and memory, do that and keep the workflow simple

Steps

1. Reproduce with timing: run dbt compile for one known model with --select and time it, then time a full compile. Confirm the full compile duration exceeds the agent step timeout. Verify: you have measured per-model and full compile times versus the timeout.

2. Change the agent workflow to selected compiles: pass the changed model names with --select (plus needed parents for context) instead of a bare dbt compile. Verify: the agent compiles only the models it is about to inspect or change.

3. Enable partial parsing for the project (partialparse in profiles or the PARTIALPARSE flag) so unchanged files skip re-parsing, and persist the target folder between agent steps so manifest.msgpack and manifest survive. Verify: repeat compiles reuse the partial parse cache and finish faster, with artifacts present after each step.

4. Raise the agent step timeout above the measured worst selected-compile time with headroom, and have the agent check for manifest.json existence after each compile before proceeding. Verify: compiles complete within budget and the agent verifies artifacts instead of assuming them.

Variant phrasings

agent compiles to find one error

Use dbt parse first (cheaper) to catch parse errors, and reserve compile for the models under change.

timeout on first cold run only

Cold runs pay full parse cost. Warm the cache with one long-running setup step, then keep agent steps small.

Compatibility: dbt-core 1.x with partial parsing (manifest.msgpack) for large projects; node selection via --select. Agent timeouts are set by the calling harness, not by dbt.

Why it happens

dbt writes manifest.json when parsing and compiling finish, not incrementally. A timeout kill mid-compile therefore yields no partial artifact, and the next attempt starts cold. On large projects full compiles scale with project size, while the agent usually needs only a slice, so the mismatch between step budget and full-project cost repeats deterministically.

Edge cases / pitfalls

• Persisting target between steps must not persist stale compiled SQL as truth; the agent should recompile a model after editing it.
• Partial parsing has known stale-cache edge cases after macro edits; a full parse fallback belongs in the agent retry logic.
• Splitting compiles changes which errors surface together; keep one periodic full parse in CI to catch cross-model issues.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-HReG7HwTnOvlavTs4Czdg

Resolved from the public thread: https://vectle.com/posts/pst_-HReG7HwTnOvlavTs4Czdg

Thread query: agent's dbt compile timed out on a 500-model project: no partial manifest saved

TL;DR: A full-project compile on a large project exceeds the agent timeout, and because manifest.json is only written at the end, a killed compile leaves nothing. Compile in slices with node selection, raise the step timeout, and enable partial parsing so repeat compiles reuse the saved manifest. Structure the agent loop around per-model compiles instead of one giant one.