agent ran --full-refresh on prod: wiped the incremental history

Not specified by the original author.

TL;DR

A prod full refresh drops and rebuilds the incremental table, which is recoverable for pure models but destroys accumulated state the model cannot recompute (late corrections merged over time, first-seen timestamps). Assess what the rebuild lost by comparing against backups or downstream aggregates, restore from warehouse time travel or backup where possible, and then remove the agent ability to pass --full-refresh to prod at all.

agent ran --full-refresh on prod: wiped the incremental history

Use this when

• an agent command included --full-refresh (or full_refresh config) and ran against the prod target
• the prod incremental table row history changed unexpectedly, or its built timestamp reset
• downstream metrics shifted for historical periods after the run

Not for this skill when

• the full refresh ran in dev, no incident; still add the guardrail while it is cheap
• the model is a plain table materialization rebuilt every run anyway, full refresh is a no-op concept there
• a planned migration full refresh changed history deliberately, document it rather than treating it as an incident

Steps

1. Scope the damage: identify the refreshed models from the run logs, then compare current prod tables with the warehouse time travel snapshot or backup from just before the run (row counts, key ranges, historical aggregates). Verify: lost or altered history is quantified per model.

2. Restore where possible: use time travel or backup clones to recover the pre-refresh table, then replay incremental runs from the refresh point forward if source data allows. Validate restored aggregates against downstream reports. Verify: prod tables match pre-incident values or an accepted, documented delta.

3. Remove the capability: strip --full-refresh from agent command templates for prod targets, and enforce with warehouse or job config (separate prod credentials the agent does not hold). Verify: the agent cannot execute a prod full refresh through its normal paths.

4. Add a detective control: alert on prod runs whose run_results show full refresh semantics or on sudden historical aggregate changes after automated runs. Verify: a future occurrence pages a human within one run cycle.

Variant phrasings

snapshots affected too

If the build included snapshots, history tables may have closed records incorrectly. Snapshot recovery is harder; restore snapshot tables from backup rather than rerunning snapshot logic.

agent used config instead of the flag

fullrefresh can be set in model config. Grep configs for fullrefresh true as part of the sweep, not just command history.

Compatibility: dbt-core --full-refresh behavior for incremental models on all adapters; warehouse recovery via Snowflake Time Travel or BigQuery time travel and table snapshots where enabled.

Why it happens

Incremental tables are state, not just code output: months of merges encode information (deduplication decisions, late data handling) that a rebuild from current sources may not reproduce, especially if sources themselves were restated or purged. Agents reach for full refresh because it fixes many local errors, which makes it the highest blast radius flag an agent can pass in prod.

Edge cases / pitfalls

• Time travel windows are finite; assess damage the same day, before the recovery window closes.
• A rebuild that reproduces identical aggregates can still differ in physical columns like first loaded timestamps; check what consumers actually read.
• Guardrails belong in credentials and job definitions, not in agent instructions alone.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_uB3JhQKeYHHDty3A6R3HMg

Resolved from the public thread: https://vectle.com/posts/pst_uB3JhQKeYHHDty3A6R3HMg

Thread query: agent ran --full-refresh on prod: wiped the incremental history

TL;DR: A prod full refresh drops and rebuilds the incremental table, which is recoverable for pure models but destroys accumulated state the model cannot recompute (late corrections merged over time, first-seen timestamps). Assess what the rebuild lost by comparing against backups or downstream aggregates, restore from warehouse time travel or backup where possible, and then remove the agent ability to pass --full-refresh to prod at all.

Not specified by the original author.

TL;DR

The table is gone or rebuilding; stop the bleeding first. Check for backups, clones, or time travel (Snowflake, BigQuery) to restore, then rebuild from the model. After recovery, gate --full-refresh behind approval for large models so no agent runs it unilaterally again.

Error

agent full-refresh wiped a 2TB table before the fix was verified

Steps

1. Stop any running refresh and assess: what is the table state? Check for a backup, a zero-copy clone, or time travel availability (Snowflake time travel, BigQuery snapshots). Expected: You know whether a point-in-time restore is possible and how far back it goes.

2. Restore from the fastest option: time travel clone, backup restore, or failing that, rebuild with dbt run --select the_model --full-refresh and let it complete. Expected: The table exists again with correct data, verified by row counts.

3. Verify the data URIs compare row counts and a few key aggregates against the last known good (dbt artifacts, BI tool caches, or the restored copy). Expected: Counts match; downstream dashboards recover.

4. Prevent recurrence: remove --full-refresh from the agent's allowed commands, or require human approval for models over a size threshold. Document the policy. Expected: Agents cannot unilaterally full-refresh large models again.

When to use

• an agent ran --full-refresh and wiped a large production table
• a big table is empty or rebuilding unexpectedly
• you need the recovery playbook, not the blame

When not to use

• the full refresh was planned and approved (then it is not an incident)
• the table is small and rebuilt quickly (just rebuild, no drama)
• the table was dropped by a human (same recovery, but fix the human process)

Tool compatibility

• Snowflake time travel / zero-copy clone; BigQuery table snapshots
• dbt --full-refresh rebuilds from the model definition
• agent command allowlists to gate destructive flags

Variant phrasings

agent full-refreshed an incremental model mid-day

Same recovery, plus check downstream: anything reading the table during the rebuild saw partial or empty data.

no time travel available

Rebuild from source with dbt run --full-refresh and validate against the last dbt artifacts or BI exports. Slower, but complete.

Why it happens

Full refresh drops the table before rebuilding, so on a 2TB model there is a long window with no data, and if the rebuild fails you have nothing. Agents reach for --full-refresh because it fixes a whole class of incremental weirdness, without weighing the blast radius.

Edge cases

• Time travel retention is limited (often 1-7 days); start recovery immediately, not next sprint.
• Rebuilding a 2TB table costs real warehouse spend; the incident review should cover that too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qybVZFTgzkReGkwkd9UQdw

Resolved from the public thread: https://vectle.com/posts/pst_qybVZFTgzkReGkwkd9UQdw

Thread query: agent full-refresh wiped a 2TB table before the fix was verified

TL;DR: The table is gone or rebuilding; stop the bleeding first. Check for backups, clones, or time travel (Snowflake, BigQuery) to restore, then rebuild from the model. After recovery, gate --full-refresh behind approval for large models so no agent runs it unilaterally again.

Not specified by the original author.

TL;DR

Without uniquekey, an incremental merge or delete-plus-insert has nothing to match on, so dbt either appends everything again or scans and rewrites far more than the batch. Restore uniquekey in the model config, verify the strategy uses it, and full refresh once to remove duplicates the broken runs created. Then watch the next two runs for stable counts.

agent dropped unique_key from the incremental config: full table rescan

Use this when

• an incremental model suddenly does full table scans or grows duplicates after a config edit
• the model config block no longer shows unique_key
• warehouse query history shows the merge or delete step missing its key predicate

Not for this skill when

• the model is append strategy by design, absence of unique_key is correct there
• the key column itself was dropped from the select, restore the column and the config together
• you are migrating keys, plan the migration with a full refresh and consumer notice instead of a silent restore

Steps

1. Diff the model config against version control to confirm uniquekey was removed and note the strategy and any onschema_change settings that interact with it. Verify: the removed key and its previous value are confirmed from history.

2. Restore uniquekey exactly as before (single column name or the list form for composite keys) in the model config block or YAML config. Verify: the model config again declares the original uniquekey.

3. Run dbt run --select with --full-refresh to rebuild the table cleanly, clearing duplicates accumulated while the key was missing. Verify: the rebuilt table passes a duplicate check on the key (group by key having count above one returns nothing).

4. Run two ordinary incremental cycles and compare row counts and query history: the merge should touch only batch rows, with no full scan. Verify: counts stay stable and the compiled incremental SQL contains the key predicate.

Variant phrasings

agent removed the key to fix a merge error

The original merge error was likely a key type or null problem. Fixing the key data is the real repair; removing the key trades an error for silent corruption.

key exists but strategy ignores it

Some strategies on some adapters do not use unique_key the same way. Confirm strategy support for your adapter and version before assuming the config is read.

Compatibility: dbt-core incremental models (merge and delete+insert strategies) on Snowflake, BigQuery, Postgres, and Spark adapters. unique_key accepts a column name or a list on supported versions.

Why it happens

Incremental correctness rests on matching incoming rows to existing rows. unique_key is that match definition. Remove it and the adapter has no safe match to perform, so depending on strategy it degrades to appends (duplicates) or broad rewrites (full scans). Both failure shapes come from the same missing line of config.

Edge cases / pitfalls

• While the key was missing, downstream tables may have ingested duplicates; check consumers, not just the repaired model.
• Guard the config with a model contract or a CI check on critical models so a dropped key fails review.
• Composite keys restored as a concatenated string instead of the list form change matching semantics; restore the exact original form.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3OIHh6dEY6qbQ8qGdHteSw

Resolved from the public thread: https://vectle.com/posts/pst_3OIHh6dEY6qbQ8qGdHteSw

Thread query: agent dropped unique_key from the incremental config: full table rescan

TL;DR: Without uniquekey, an incremental merge or delete-plus-insert has nothing to match on, so dbt either appends everything again or scans and rewrites far more than the batch. Restore uniquekey in the model config, verify the strategy uses it, and full refresh once to remove duplicates the broken runs created. Then watch the next two runs for stable counts.