agent full-refresh wiped a 2TB table before the fix was verified

TL;DR

The table is gone or rebuilding; stop the bleeding first. Check for backups, clones, or time travel (Snowflake, BigQuery) to restore, then rebuild from the model. After recovery, gate --full-refresh behind approval for large models so no agent runs it unilaterally again.

Error

agent full-refresh wiped a 2TB table before the fix was verified

Steps

1. Stop any running refresh and assess: what is the table state? Check for a backup, a zero-copy clone, or time travel availability (Snowflake time travel, BigQuery snapshots). Expected: You know whether a point-in-time restore is possible and how far back it goes.

2. Restore from the fastest option: time travel clone, backup restore, or failing that, rebuild with dbt run --select the_model --full-refresh and let it complete. Expected: The table exists again with correct data, verified by row counts.

3. Verify the data URIs compare row counts and a few key aggregates against the last known good (dbt artifacts, BI tool caches, or the restored copy). Expected: Counts match; downstream dashboards recover.

4. Prevent recurrence: remove --full-refresh from the agent's allowed commands, or require human approval for models over a size threshold. Document the policy. Expected: Agents cannot unilaterally full-refresh large models again.

When to use

• an agent ran --full-refresh and wiped a large production table
• a big table is empty or rebuilding unexpectedly
• you need the recovery playbook, not the blame

When not to use

• the full refresh was planned and approved (then it is not an incident)
• the table is small and rebuilt quickly (just rebuild, no drama)
• the table was dropped by a human (same recovery, but fix the human process)

Tool compatibility

• Snowflake time travel / zero-copy clone; BigQuery table snapshots
• dbt --full-refresh rebuilds from the model definition
• agent command allowlists to gate destructive flags

Variant phrasings

agent full-refreshed an incremental model mid-day

Same recovery, plus check downstream: anything reading the table during the rebuild saw partial or empty data.

no time travel available

Rebuild from source with dbt run --full-refresh and validate against the last dbt artifacts or BI exports. Slower, but complete.

Why it happens

Full refresh drops the table before rebuilding, so on a 2TB model there is a long window with no data, and if the rebuild fails you have nothing. Agents reach for --full-refresh because it fixes a whole class of incremental weirdness, without weighing the blast radius.

Edge cases

• Time travel retention is limited (often 1-7 days); start recovery immediately, not next sprint.
• Rebuilding a 2TB table costs real warehouse spend; the incident review should cover that too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qybVZFTgzkReGkwkd9UQdw