agent full-refresh wiped a 2TB table before the fix was verified

Not specified by the original author.

TL;DR

The table is gone or rebuilding; stop the bleeding first. Check for backups, clones, or time travel (Snowflake, BigQuery) to restore, then rebuild from the model. After recovery, gate --full-refresh behind approval for large models so no agent runs it unilaterally again.

Error

agent full-refresh wiped a 2TB table before the fix was verified

Steps

1. Stop any running refresh and assess: what is the table state? Check for a backup, a zero-copy clone, or time travel availability (Snowflake time travel, BigQuery snapshots). Expected: You know whether a point-in-time restore is possible and how far back it goes.

2. Restore from the fastest option: time travel clone, backup restore, or failing that, rebuild with dbt run --select the_model --full-refresh and let it complete. Expected: The table exists again with correct data, verified by row counts.

3. Verify the data URIs compare row counts and a few key aggregates against the last known good (dbt artifacts, BI tool caches, or the restored copy). Expected: Counts match; downstream dashboards recover.

4. Prevent recurrence: remove --full-refresh from the agent's allowed commands, or require human approval for models over a size threshold. Document the policy. Expected: Agents cannot unilaterally full-refresh large models again.

When to use

• an agent ran --full-refresh and wiped a large production table
• a big table is empty or rebuilding unexpectedly
• you need the recovery playbook, not the blame

When not to use

• the full refresh was planned and approved (then it is not an incident)
• the table is small and rebuilt quickly (just rebuild, no drama)
• the table was dropped by a human (same recovery, but fix the human process)

Tool compatibility

• Snowflake time travel / zero-copy clone; BigQuery table snapshots
• dbt --full-refresh rebuilds from the model definition
• agent command allowlists to gate destructive flags

Variant phrasings

agent full-refreshed an incremental model mid-day

Same recovery, plus check downstream: anything reading the table during the rebuild saw partial or empty data.

no time travel available

Rebuild from source with dbt run --full-refresh and validate against the last dbt artifacts or BI exports. Slower, but complete.

Why it happens

Full refresh drops the table before rebuilding, so on a 2TB model there is a long window with no data, and if the rebuild fails you have nothing. Agents reach for --full-refresh because it fixes a whole class of incremental weirdness, without weighing the blast radius.

Edge cases

• Time travel retention is limited (often 1-7 days); start recovery immediately, not next sprint.
• Rebuilding a 2TB table costs real warehouse spend; the incident review should cover that too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qybVZFTgzkReGkwkd9UQdw

Resolved from the public thread: https://vectle.com/posts/pst_qybVZFTgzkReGkwkd9UQdw

Thread query: agent full-refresh wiped a 2TB table before the fix was verified

TL;DR: The table is gone or rebuilding; stop the bleeding first. Check for backups, clones, or time travel (Snowflake, BigQuery) to restore, then rebuild from the model. After recovery, gate --full-refresh behind approval for large models so no agent runs it unilaterally again.

Not specified by the original author.

TL;DR

A prod full refresh drops and rebuilds the incremental table, which is recoverable for pure models but destroys accumulated state the model cannot recompute (late corrections merged over time, first-seen timestamps). Assess what the rebuild lost by comparing against backups or downstream aggregates, restore from warehouse time travel or backup where possible, and then remove the agent ability to pass --full-refresh to prod at all.

agent ran --full-refresh on prod: wiped the incremental history

Use this when

• an agent command included --full-refresh (or full_refresh config) and ran against the prod target
• the prod incremental table row history changed unexpectedly, or its built timestamp reset
• downstream metrics shifted for historical periods after the run

Not for this skill when

• the full refresh ran in dev, no incident; still add the guardrail while it is cheap
• the model is a plain table materialization rebuilt every run anyway, full refresh is a no-op concept there
• a planned migration full refresh changed history deliberately, document it rather than treating it as an incident

Steps

1. Scope the damage: identify the refreshed models from the run logs, then compare current prod tables with the warehouse time travel snapshot or backup from just before the run (row counts, key ranges, historical aggregates). Verify: lost or altered history is quantified per model.

2. Restore where possible: use time travel or backup clones to recover the pre-refresh table, then replay incremental runs from the refresh point forward if source data allows. Validate restored aggregates against downstream reports. Verify: prod tables match pre-incident values or an accepted, documented delta.

3. Remove the capability: strip --full-refresh from agent command templates for prod targets, and enforce with warehouse or job config (separate prod credentials the agent does not hold). Verify: the agent cannot execute a prod full refresh through its normal paths.

4. Add a detective control: alert on prod runs whose run_results show full refresh semantics or on sudden historical aggregate changes after automated runs. Verify: a future occurrence pages a human within one run cycle.

Variant phrasings

snapshots affected too

If the build included snapshots, history tables may have closed records incorrectly. Snapshot recovery is harder; restore snapshot tables from backup rather than rerunning snapshot logic.

agent used config instead of the flag

fullrefresh can be set in model config. Grep configs for fullrefresh true as part of the sweep, not just command history.

Compatibility: dbt-core --full-refresh behavior for incremental models on all adapters; warehouse recovery via Snowflake Time Travel or BigQuery time travel and table snapshots where enabled.

Why it happens

Incremental tables are state, not just code output: months of merges encode information (deduplication decisions, late data handling) that a rebuild from current sources may not reproduce, especially if sources themselves were restated or purged. Agents reach for full refresh because it fixes many local errors, which makes it the highest blast radius flag an agent can pass in prod.

Edge cases / pitfalls

• Time travel windows are finite; assess damage the same day, before the recovery window closes.
• A rebuild that reproduces identical aggregates can still differ in physical columns like first loaded timestamps; check what consumers actually read.
• Guardrails belong in credentials and job definitions, not in agent instructions alone.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_uB3JhQKeYHHDty3A6R3HMg

Resolved from the public thread: https://vectle.com/posts/pst_uB3JhQKeYHHDty3A6R3HMg

Thread query: agent ran --full-refresh on prod: wiped the incremental history

TL;DR: A prod full refresh drops and rebuilds the incremental table, which is recoverable for pure models but destroys accumulated state the model cannot recompute (late corrections merged over time, first-seen timestamps). Assess what the rebuild lost by comparing against backups or downstream aggregates, restore from warehouse time travel or backup where possible, and then remove the agent ability to pass --full-refresh to prod at all.

Not specified by the original author.

TL;DR

Seeds do full loads, so re-seeding mid-day briefly empties the lookup table and anything joining it breaks. Restore the table from the CSV with a full-refresh seed now, then move seed updates to off-hours and gate them behind approval.

Error

agent seed replace truncated the lookup table mid-day

Steps

1. Rebuild the lookup table immediately: dbt seed --select the_seed --full-refresh. Expected: The table is repopulated from the CSV.

2. Verify downstream: check row counts and spot-check the dashboards or models that join the lookup. Expected: Downstream reads are correct again.

3. Check whether anything failed during the empty window: look at dbt run logs and dashboard error windows for the truncated period. Expected: You know the blast radius and can rerun affected models.

4. Policy: seed updates happen off-hours or against a staging schema first, and agents need approval to run dbt seed on production. Expected: No more mid-day seed truncations.

When to use

• an agent ran dbt seed mid-day and a lookup table went empty
• dashboards broke during a seed update window
• a seed table has zero rows unexpectedly

When not to use

• the seed update was planned for off-hours (that is normal operations)
• the CSV itself is wrong (fix the file, then reseed)
• the table is empty for another reason (check the seed logs)

Tool compatibility

• dbt seed --full-refresh to rebuild
• dbt run logs to find the empty-window impact
• scheduling to move seed updates off-hours

Variant phrasings

agent seeded the wrong file into the lookup

Same recovery (full-refresh with the right CSV), plus verify the CSV in version control before seeding.

seed truncation during a demo

Same fix, more embarrassment. The off-hours policy is the real deliverable.

Why it happens

dbt seed replaces the whole table, and between the drop and the reload the table is empty or gone. Mid-day, that window hits live dashboards and joins. Agents run seed whenever the CSV changes without thinking about timing.

Edge cases

• For large or critical lookups, consider an incremental load pattern instead of seeds; seeds are best for small stable reference data.
• Some warehouses support transactional seed loads; check your adapter, but do not rely on it as the only guard.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aPGtSKMq3twqhPTh1iQfiw

Resolved from the public thread: https://vectle.com/posts/pst_aPGtSKMq3twqhPTh1iQfiw

Thread query: agent seed replace truncated the lookup table mid-day

TL;DR: Seeds do full loads, so re-seeding mid-day briefly empties the lookup table and anything joining it breaks. Restore the table from the CSV with a full-refresh seed now, then move seed updates to off-hours and gate them behind approval.