agent ran --full-refresh on prod: wiped the incremental history

TL;DR

A prod full refresh drops and rebuilds the incremental table, which is recoverable for pure models but destroys accumulated state the model cannot recompute (late corrections merged over time, first-seen timestamps). Assess what the rebuild lost by comparing against backups or downstream aggregates, restore from warehouse time travel or backup where possible, and then remove the agent ability to pass --full-refresh to prod at all.

agent ran --full-refresh on prod: wiped the incremental history

Use this when

• an agent command included --full-refresh (or full_refresh config) and ran against the prod target
• the prod incremental table row history changed unexpectedly, or its built timestamp reset
• downstream metrics shifted for historical periods after the run

Not for this skill when

• the full refresh ran in dev, no incident; still add the guardrail while it is cheap
• the model is a plain table materialization rebuilt every run anyway, full refresh is a no-op concept there
• a planned migration full refresh changed history deliberately, document it rather than treating it as an incident

Steps

1. Scope the damage: identify the refreshed models from the run logs, then compare current prod tables with the warehouse time travel snapshot or backup from just before the run (row counts, key ranges, historical aggregates). Verify: lost or altered history is quantified per model.

2. Restore where possible: use time travel or backup clones to recover the pre-refresh table, then replay incremental runs from the refresh point forward if source data allows. Validate restored aggregates against downstream reports. Verify: prod tables match pre-incident values or an accepted, documented delta.

3. Remove the capability: strip --full-refresh from agent command templates for prod targets, and enforce with warehouse or job config (separate prod credentials the agent does not hold). Verify: the agent cannot execute a prod full refresh through its normal paths.

4. Add a detective control: alert on prod runs whose run_results show full refresh semantics or on sudden historical aggregate changes after automated runs. Verify: a future occurrence pages a human within one run cycle.

Variant phrasings

snapshots affected too

If the build included snapshots, history tables may have closed records incorrectly. Snapshot recovery is harder; restore snapshot tables from backup rather than rerunning snapshot logic.

agent used config instead of the flag

fullrefresh can be set in model config. Grep configs for fullrefresh true as part of the sweep, not just command history.

Compatibility: dbt-core --full-refresh behavior for incremental models on all adapters; warehouse recovery via Snowflake Time Travel or BigQuery time travel and table snapshots where enabled.

Why it happens

Incremental tables are state, not just code output: months of merges encode information (deduplication decisions, late data handling) that a rebuild from current sources may not reproduce, especially if sources themselves were restated or purged. Agents reach for full refresh because it fixes many local errors, which makes it the highest blast radius flag an agent can pass in prod.

Edge cases / pitfalls

• Time travel windows are finite; assess damage the same day, before the recovery window closes.
• A rebuild that reproduces identical aggregates can still differ in physical columns like first loaded timestamps; check what consumers actually read.
• Guardrails belong in credentials and job definitions, not in agent instructions alone.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_uB3JhQKeYHHDty3A6R3HMg