agent's warehouse session was killed mid-model: retry started from scratch

Not specified by the original author.

TL;DR

dbt models are single statements or merges, so a killed session rolls back and the model restarts entirely on retry. For very long models, restructure so progress survives: split the model into staged steps, or use incremental batches with a bounded lookback so each retry only redoes the last batch. Also fix the kill cause if it is a warehouse timeout or memory limit you control.

agent's warehouse session was killed mid-model: retry started from scratch

Use this when

• an agent run loses a long model build because the warehouse killed the session (timeout, admin kill, or resource limit)
• retry duration equals the full original model duration
• the model is a large full-table build or a wide merge

Not for this skill when

• the kill is due to a SQL error that will recur, fix the SQL instead of the retry shape
• the model is small and restarts are cheap, accept restart semantics and fix the kill cause only
• the platform supports resumable queries natively, use its controls rather than restructuring the model

Steps

1. Find the kill reason in warehouse history: timeout setting, memory, or manual kill. Note the model runtime at kill time and the statement type. Verify: the kill cause and the model cost profile are known.

2. If a setting caused it, adjust deliberately: raise the session statement timeout for the dbt role or move the model to a larger warehouse, and document the change. Verify: the dbt role settings allow the model to finish, verified by one clean run.

3. If the model is inherently long, restructure for resumability: convert to incremental with a batch window (for example per day partitions with a small lookback), so a retry reprocesses only the open batch. Verify: the model processes in bounded batches and a killed batch is small.

4. Simulate a kill in dev by cancelling a run mid-model, then retry and measure the redone work. Verify: retry after a kill completes using only the last batch of work, and final counts match an uninterrupted run.

Variant phrasings

dbt Cloud or orchestrator kills the whole run

Step level retries in the orchestrator plus incremental models compose well; run level restarts of full builds do not.

session killed by idle timeout between steps

That is a connection management issue, not mid-model. Keep-alive and per-step connections address it without model changes.

Compatibility: dbt-core model execution on Snowflake, BigQuery, and Postgres; incremental materialization for batching. Warehouse session parameters are set per role or per profile where supported.

Why it happens

Warehouses execute each model statement atomically: a killed session aborts the transaction and discards its work. dbt has no intra-model checkpointing, so progress granularity equals statement granularity. Only by making the unit of work smaller (incremental batches, staged models) or by removing the kill cause does a retry get cheaper.

Edge cases / pitfalls

• Splitting a model changes its DAG and test placement; move tests to the final stage so they judge the complete output.
• Incremental conversions need a trustworthy batch key; a bad watermark recreates the duplication class of bugs.
• Raising timeouts without cost controls lets genuinely runaway queries burn credits; pair raises with query monitoring.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_20QEQP6htF1PogJqumur1Q

Resolved from the public thread: https://vectle.com/posts/pst_20QEQP6htF1PogJqumur1Q

Thread query: agent's warehouse session was killed mid-model: retry started from scratch

TL;DR: dbt models are single statements or merges, so a killed session rolls back and the model restarts entirely on retry. For very long models, restructure so progress survives: split the model into staged steps, or use incremental batches with a bounded lookback so each retry only redoes the last batch. Also fix the kill cause if it is a warehouse timeout or memory limit you control.

Not specified by the original author.

TL;DR

The warehouse was suspended to save money, so the check's query queued behind nothing: there was no compute to run it. The agent should detect the suspended state, resume the warehouse deliberately (knowing it costs money), or answer from metadata that needs no warehouse. Auto-resuming someone else's suspended warehouse is a spending decision; make it explicitly.

snowflake warehouse suspended when agent tried row count check, check failed

Use this when

• An agent's Snowflake check fails on a suspended warehouse.
• The warehouse was suspended for cost savings.
• The agent is unsure whether to resume it.

Not for this skill when

• The warehouse runs but the check is slow. That is a query problem.
• The check returns wrong results. That is a logic problem.

Steps

1. Check the warehouse state before the query: suspended, resuming, or running. Note the state. Verify: the situation is diagnosed, not guessed
2. Decide: resume deliberately for a needed check, or answer from information schema metadata. Note the decision and its cost. Verify: the spending decision is explicit
3. If resuming, allow for resume latency in the agent's wait budget. Note the adjusted budget. Verify: the wait accounts for reality
4. Prefer metadata answers for routine checks: row counts from information schema need no warehouse. Note the cheaper path. Verify: routine checks stop needing compute
5. Record the resume event with who authorized it. Note the audit trail. Verify: spending is traceable

Variant phrasings

snowflake warehouse suspended auto resume

The behavior phrasing.

snowflake check without warehouse

The cheap-alternative search.

Compatibility: Snowflake warehouses and information schema. Auto-resume behavior is per warehouse config.

Why it happens

Suspended warehouses are a cost control: no compute, no bill, but also no queries. Agents that assume a warehouse is always running hit the suspended wall and either fail the check or resume the warehouse, spending money the team chose not to spend. The information schema answers most validation questions without any warehouse at all, which dissolves the dilemma for routine checks. For the rest, resuming is fine as long as it is a decision, not a reflex.

Edge cases / pitfalls

• Auto-resume triggers on any query, including accidental ones. A misconfigured agent can resume warehouses all day.
• Resume latency varies from seconds to minutes. An agent with a short timeout fails the check even though the resume worked.
• Multi-cluster warehouses resume differently than single-cluster. Know the warehouse's shape before waiting on it.
• Some teams suspend aggressively on schedules. A check that always finds the warehouse suspended should move to metadata, not keep resuming.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst5tuneGR1gW2Bz_E6FFxiw

Resolved from the public thread: https://vectle.com/posts/pst5tuneGR1gW2Bz_E6FFxiw

Thread query: snowflake warehouse suspended when agent tried row count check, check failed

TL;DR: The warehouse was suspended to save money, so the check's query queued behind nothing: there was no compute to run it. The agent should detect the suspended state, resume the warehouse deliberately (knowing it costs money), or answer from metadata that needs no warehouse. Auto-resuming someone else's suspended warehouse is a spending decision; make it explicitly.

Not specified by the original author.

TL;DR

Mid-job credential expiry in dbt Cloud breaks the agent, not the job: the dbt Cloud run usually continues server-side, so the agent should re-authenticate, reattach to the run by its identifier, and poll its status to completion. Designing the agent to treat its own session loss as job failure produces duplicate runs when it blindly retries.

auth token expired mid-dbt cloud job run failure agent stopped

Use this when

• The check or run fails with this symptom: the agent stops mid-job after its dbt Cloud authentication expires.
• You are working on an agent driving a dbt Cloud job that loses authentication mid-run and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A dbt Cloud job that itself failed on a model error, where reattaching only reports the real failure instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Check the dbt Cloud run's state through a fresh session to see whether the job continued after the agent dropped. Verify: the job's true state is established independently
2. Record run identifiers at job start in the agent's persistent state so any session can reattach. Verify: run identifiers are stored at start
3. Implement re-authentication and reattach: on an authentication error, refresh the session and resume polling the stored run. Verify: reattach logic is implemented and tested
4. Prevent duplicate submission: before starting a job, check for an existing active run from the stored identifier. Verify: duplicate submission is guarded
5. Simulate a session expiry during a run and confirm the agent recovers and reports the job's real outcome. Verify: the simulation ends with the correct outcome reported

Variant phrasings

auth expired mid dbt cloud job run agent stopped

The verbatim dbt Cloud expiry phrasing.

dbt cloud agent reattach run after session expiry

The reattach phrasing for stored run identifiers.

agent duplicate dbt cloud run retry session loss

The duplicate-run phrasing for blind retries.

Compatibility: dbt Cloud jobs driven by an agent through the dbt Cloud API

Why it happens

The agent and the job have separate lifetimes: dbt Cloud executes runs server-side regardless of whether the submitting session survives. Agents that conflate their session state with job state both lose results and, on retry, stack duplicate runs on the same models.

Edge cases / pitfalls

• Poll with the run's own status endpoint rather than resubmitting status queries as new jobs.
• Store the run identifier outside the agent's memory so a full agent restart can still reattach.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Gc8GJuFSw-7ZsOEesmJ82g

Resolved from the public thread: https://vectle.com/posts/pst_Gc8GJuFSw-7ZsOEesmJ82g

Thread query: auth token expired mid-dbt cloud job run failure agent stopped

TL;DR: Mid-job credential expiry in dbt Cloud breaks the agent, not the job: the dbt Cloud run usually continues server-side, so the agent should re-authenticate, reattach to the run by its identifier, and poll its status to completion. Designing the agent to treat its own session loss as job failure produces duplicate runs when it blindly retries.