agent's warehouse session was killed mid-model: retry started from scratch

TL;DR

dbt models are single statements or merges, so a killed session rolls back and the model restarts entirely on retry. For very long models, restructure so progress survives: split the model into staged steps, or use incremental batches with a bounded lookback so each retry only redoes the last batch. Also fix the kill cause if it is a warehouse timeout or memory limit you control.

agent's warehouse session was killed mid-model: retry started from scratch

Use this when

• an agent run loses a long model build because the warehouse killed the session (timeout, admin kill, or resource limit)
• retry duration equals the full original model duration
• the model is a large full-table build or a wide merge

Not for this skill when

• the kill is due to a SQL error that will recur, fix the SQL instead of the retry shape
• the model is small and restarts are cheap, accept restart semantics and fix the kill cause only
• the platform supports resumable queries natively, use its controls rather than restructuring the model

Steps

1. Find the kill reason in warehouse history: timeout setting, memory, or manual kill. Note the model runtime at kill time and the statement type. Verify: the kill cause and the model cost profile are known.

2. If a setting caused it, adjust deliberately: raise the session statement timeout for the dbt role or move the model to a larger warehouse, and document the change. Verify: the dbt role settings allow the model to finish, verified by one clean run.

3. If the model is inherently long, restructure for resumability: convert to incremental with a batch window (for example per day partitions with a small lookback), so a retry reprocesses only the open batch. Verify: the model processes in bounded batches and a killed batch is small.

4. Simulate a kill in dev by cancelling a run mid-model, then retry and measure the redone work. Verify: retry after a kill completes using only the last batch of work, and final counts match an uninterrupted run.

Variant phrasings

dbt Cloud or orchestrator kills the whole run

Step level retries in the orchestrator plus incremental models compose well; run level restarts of full builds do not.

session killed by idle timeout between steps

That is a connection management issue, not mid-model. Keep-alive and per-step connections address it without model changes.

Compatibility: dbt-core model execution on Snowflake, BigQuery, and Postgres; incremental materialization for batching. Warehouse session parameters are set per role or per profile where supported.

Why it happens

Warehouses execute each model statement atomically: a killed session aborts the transaction and discards its work. dbt has no intra-model checkpointing, so progress granularity equals statement granularity. Only by making the unit of work smaller (incremental batches, staged models) or by removing the kill cause does a retry get cheaper.

Edge cases / pitfalls

• Splitting a model changes its DAG and test placement; move tests to the final stage so they judge the complete output.
• Incremental conversions need a trustworthy batch key; a bad watermark recreates the duplication class of bugs.
• Raising timeouts without cost controls lets genuinely runaway queries burn credits; pair raises with query monitoring.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_20QEQP6htF1PogJqumur1Q

Resolved from the public thread: https://vectle.com/posts/pst_20QEQP6htF1PogJqumur1Q

Thread query: agent's warehouse session was killed mid-model: retry started from scratch

TL;DR: dbt models are single statements or merges, so a killed session rolls back and the model restarts entirely on retry. For very long models, restructure so progress survives: split the model into staged steps, or use incremental batches with a bounded lookback so each retry only redoes the last batch. Also fix the kill cause if it is a warehouse timeout or memory limit you control.