agent re-ran dbt build in prod to verify: double-counted events in incremental model

TL;DR

Re-running the incremental build re-processed the same window twice because the incremental filter did not exclude already-loaded rows. Dedupe the model to the event grain, fix the incremental predicate so reruns are idempotent, and backfill the affected window cleanly.

Error

agent re-ran dbt build in prod to verify: double-counted events in incremental model

Steps

1. Confirm the double count: compare the model's row counts per event day against the source; the affected window shows roughly 2x rows. Expected: You identify the duplicated date range precisely.

2. Inspect the model's incremental logic: the isincremental() block should filter to new rows only (e.g. eventdate > (select max(event_date) from this)). Find the gap that let old rows through. Expected: You find the predicate flaw, e.g. comparing on the wrong column or no filter at all.

3. Fix the predicate, then dedupe the table: rebuild the affected window with a full-refresh limited to that period, or delete and re-insert the duplicated range. Expected: The table has one row per event again; counts match the source.

4. Add a uniqueness test on the event grain key so the next double-load fails loudly. Expected: dbt test guards the grain going forward.

When to use

• an incremental model double counts after an agent re-ran dbt build in prod
• re-running a build changes totals (non-idempotent incremental)
• the duplicated window matches a re-run timestamp

When not to use

• the source itself sent duplicates (that is upstream, fix the extraction)
• the grain changed (that is a fanout, different fix)
• a one-time backfill intentionally reloaded history (then dedupe as part of the backfill plan)

Tool compatibility

• dbt incremental materialization; is_incremental() predicate
• dbt build --select with --full-refresh for window rebuilds
• uniqueness tests to lock the event grain

Variant phrasings

agent re-ran only the test, not the build

Tests do not write data, so they cannot double count. If counts changed, something rebuilt the model; check the run logs.

double count after a failed then retried build

The failed run may have committed partial rows before dying. Same fix: predicate plus dedupe of the partial window.

Why it happens

Incremental models are only safe to re-run if the incremental filter truly excludes already-loaded rows. Agents re-run builds to verify fixes without checking idempotency, and a weak predicate happily inserts the same events again.

Edge cases

• Never verify fixes by re-running incremental builds in prod; verify in dev or on a cloned schema.
• Late-arriving events need a lookback window in the predicate; too wide a lookback reintroduces dupes, size it deliberately.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IqbFnfoMX5kvEm9AIeHDGg