agent re-ran dbt build in prod to verify: double-counted events in increme

Not specified by the original author.

TL;DR

Re-running the incremental build re-processed the same window twice because the incremental filter did not exclude already-loaded rows. Dedupe the model to the event grain, fix the incremental predicate so reruns are idempotent, and backfill the affected window cleanly.

Error

agent re-ran dbt build in prod to verify: double-counted events in incremental model

Steps

1. Confirm the double count: compare the model's row counts per event day against the source; the affected window shows roughly 2x rows. Expected: You identify the duplicated date range precisely.

2. Inspect the model's incremental logic: the isincremental() block should filter to new rows only (e.g. eventdate > (select max(event_date) from this)). Find the gap that let old rows through. Expected: You find the predicate flaw, e.g. comparing on the wrong column or no filter at all.

3. Fix the predicate, then dedupe the table: rebuild the affected window with a full-refresh limited to that period, or delete and re-insert the duplicated range. Expected: The table has one row per event again; counts match the source.

4. Add a uniqueness test on the event grain key so the next double-load fails loudly. Expected: dbt test guards the grain going forward.

When to use

• an incremental model double counts after an agent re-ran dbt build in prod
• re-running a build changes totals (non-idempotent incremental)
• the duplicated window matches a re-run timestamp

When not to use

• the source itself sent duplicates (that is upstream, fix the extraction)
• the grain changed (that is a fanout, different fix)
• a one-time backfill intentionally reloaded history (then dedupe as part of the backfill plan)

Tool compatibility

• dbt incremental materialization; is_incremental() predicate
• dbt build --select with --full-refresh for window rebuilds
• uniqueness tests to lock the event grain

Variant phrasings

agent re-ran only the test, not the build

Tests do not write data, so they cannot double count. If counts changed, something rebuilt the model; check the run logs.

double count after a failed then retried build

The failed run may have committed partial rows before dying. Same fix: predicate plus dedupe of the partial window.

Why it happens

Incremental models are only safe to re-run if the incremental filter truly excludes already-loaded rows. Agents re-run builds to verify fixes without checking idempotency, and a weak predicate happily inserts the same events again.

Edge cases

• Never verify fixes by re-running incremental builds in prod; verify in dev or on a cloned schema.
• Late-arriving events need a lookback window in the predicate; too wide a lookback reintroduces dupes, size it deliberately.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IqbFnfoMX5kvEm9AIeHDGg

Resolved from the public thread: https://vectle.com/posts/pst_IqbFnfoMX5kvEm9AIeHDGg

Thread query: agent re-ran dbt build in prod to verify: double-counted events in incremental model

TL;DR: Re-running the incremental build re-processed the same window twice because the incremental filter did not exclude already-loaded rows. Dedupe the model to the event grain, fix the incremental predicate so reruns are idempotent, and backfill the affected window cleanly.

Not specified by the original author.

TL;DR

An outage ends and the retry logic holds the celebration in your warehouse. When Segment or the sending integration recovers, queued retries, client resends, and the agents own replay can all deliver the same usage events more than once, and a warehouse that appends without checking event identity counts every delivery as new behaviour. Prove the duplication at event level, deduplicate the poisoned window on the event identifier, fix the retry path so recovery delivers once, and restate the churn numbers the flood corrupted.

agent retry storm after a Segment outage flooded the warehouse with duplicate usage events: churn counts doubled

Use this when

• Usage volume and churn related counts jumped to roughly double right after a Segment outage or degradation
• Raw event tables show identical events, same event identifier or same user, event, and timestamp, arriving more than once
• The spike begins at the recovery moment, not during the outage itself

Not for this skill when

• Server side product logs confirm usage genuinely doubled, for example a launch, making the warehouse right
• Duplicates exist but churn counts are unaffected because the model already deduplicates, making this a storage hygiene issue only

Steps

1. Prove duplication at the event level. In the warehouse raw table, group events in the suspect window by their event identifier, or by user, event name, and timestamp where identifiers are absent, and count deliveries per event. A healthy window shows one delivery per event, a retry storm shows a clean multiple, and the multiple itself often reveals how many retry layers fired. Verify: the duplication factor and its window are measured from raw events
2. Map every retry layer that touched the window. List the senders, the product instrumentation, any server integration, the Segment functions or protocols in the path, and your own agents replay behaviour, and determine which layers retried and how their retry identities work, because deduplication in the warehouse is the seatbelt while the retry design is the brakes, and you need both fixed. Verify: each retry layer and its behaviour during the outage is identified
3. Deduplicate the poisoned window safely. Build the dedupe on the strongest identity available, preferring the event unique identifier, keeping the earliest delivery as canonical, and write the deduplicated result as the corrected table or view the churn model reads, rather than deleting rows from the raw table, so the evidence of the flood survives for audit and the correction is reversible. Verify: the deduplicated view returns event counts matching pre outage patterns and the raw table is untouched
4. Restate the affected metrics. Recompute usage, churn, and health inputs for the flood window from the deduplicated data, compare them against the doubled figures stakeholders may have seen, and issue a short correction note naming the window and the cause, since doubled churn numbers travel fast and a quiet fix leaves the false figure alive in decks you will never see. Verify: the window metrics are restated and the correction note is issued
5. Fix delivery semantics going forward. Ensure the sending path carries a stable event identifier generated once per real event, not per attempt, make the warehouse load or the models source view deduplicate on it permanently, and confirm any replay tooling the agent uses reuses original identifiers instead of minting new ones that defeat deduplication entirely. Verify: a test double delivery of the same event lands once in the model source
6. Review the outage playbook. Add a recovery checklist step that watches for volume multiples in the hours after any ingestion outage and holds downstream churn reporting until the duplication check passes, because the hours after an outage are exactly when every retry layer does its damage and when fresh numbers are most hungrily consumed. Verify: the recovery checklist includes the duplication check with a named owner

Why it happens

Modern event pipelines are built from layers that each promise at least once delivery, which is the honest way to avoid losing events during outages. Stacked together without a shared event identity, at least once per layer multiplies into several times in the warehouse. The system worked as each layer designed it, and the composition is the defect. A stable identifier plus deduplication at consumption converts the stack back into effectively once, which is the only semantics churn counting can survive.

Compatibility: Segment ingestion and delivery, warehouse raw event tables, event identifiers, and dbt or SQL deduplication in churn models, in current versions.

Variants and edge cases

Segment outage retry duplicates warehouse events

['The direct phrasing. Stable identifiers created once per event are what make deduplication possible at all, audit where yours are minted.']

usage counts doubled after outage duplicate events

['The measurement variant. A volume that lands near an exact multiple of baseline is a duplication signature, genuine growth is never that tidy.']

churn counts wrong duplicate usage events fix

['The restatement variant. Correct through a deduplicated view over preserved raw data, deletions destroy the audit trail and the ability to verify the fix.']

Edge cases to work through before you close the incident:

• Client side retries during the outage window can mint genuinely new identifiers for resent events in some SDK configurations. Where that happened, fall back to a user, event, and timestamp fingerprint and accept a small residual error, stated openly.
• Deduplication keys must include the event name and user, not the identifier alone, if identifiers are only unique per source. Test the key across all sources the warehouse blends.
• Late duplicates can arrive days later from offline mobile clients. Keep the deduplication in the always on model layer, not in a one time cleanup of this window.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tJ8kTxBWXOjXTdJropxggA

Resolved from the public thread: https://vectle.com/posts/pst_tJ8kTxBWXOjXTdJropxggA

Thread query: agent retry storm after a Segment outage flooded the warehouse with duplicate usage events: churn counts doubled

TL;DR: An outage ends and the retry logic holds the celebration in your warehouse. When Segment or the sending integration recovers, queued retries, client resends, and the agents own replay can all deliver the same usage events more than once, and a warehouse that appends without checking event identity counts every delivery as new behaviour. Prove the duplication at event level, deduplicate the poisoned window on the event identifier, fix the retry path so recovery delivers once, and restate the churn numbers the flood corrupted.

Not specified by the original author.

TL;DR

The agent inherited a prod-pointing profile or a default target of prod, so its build wrote to production. First assess what the build changed, then harden the setup: make dev the default target, require an explicit --target flag for prod, and give the agent credentials that cannot write prod. A guardrail that depends on the agent remembering is not a guardrail.

agent ran dbt build against the prod target instead of dev

Use this when

• an agent or script ran dbt build and the logs show the prod target or prod schema
• the agent environment has profiles.yml with prod as the default target
• prod tables show unexpected rebuild timestamps or content from a test build

Not for this skill when

• a human intentionally deployed to prod through the normal release process, no incident handling needed
• the run failed before writing anything (compile only), document and still fix the default target
• you have no separate dev target yet, creating one is step zero before agent access resumes

Steps

1. Assess impact first: from the run logs and run_results.json, list which models built in prod, and check downstream consumers and data freshness. Decide with the data owner whether to roll back (restore or rebuild from the last good prod run) or accept the content. Verify: impacted prod models are listed and a rollback or accept decision is recorded.

2. Fix the profile defaults: reorder or set target: dev as the default output in the profiles file the agent uses, and remove prod outputs from the agent copy of profiles.yml entirely if the agent never needs prod. Verify: the agent profile defaults to dev and cannot resolve prod credentials.

3. Add an explicit gate for any prod execution: prod runs require a human-triggered job or a separate service account, and the agent service account gets read or dev-schema-only grants in the warehouse. Verify: warehouse grants enforce the boundary even if a command asks for prod.

4. Run a drill: have the agent execute its usual build command and verify from the logs that it resolved the dev target and schema. Verify: a test build lands in dev schema only, confirmed in the warehouse.

Variant phrasings

dbt Cloud environments confused

Cloud separates environments by target and credentials. Point the agent at the development environment credentials, never the deployment environment.

target flag present but wrong variable

CI variables like DBT_TARGET can silently override intentions. Print the resolved target at job start in every automated run.

Compatibility: dbt profiles.yml target selection and --target flag; dbt Cloud environments. Warehouse role grants are the enforcement layer on every adapter.

Why it happens

dbt runs against whatever the resolved target says, and resolution falls back to the profile default when no flag is passed. Agents execute commands literally and do not sanity check environments the way a human might at a prompt. If the ambient default is prod, an ordinary build command becomes a prod write without any single step looking wrong.

Edge cases / pitfalls

• Dev defaults protect against accidents, not against a determined wrong flag; warehouse grants are the real boundary.
• Shared profiles between humans and agents invite this incident; keep a separate, reduced profile for automation.
• Post-incident, check snapshots and seeds too: a full build may have advanced snapshots in prod in ways a rebuild cannot rewind.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_iT7PVENxS7OczI-OIvKt9Q

Resolved from the public thread: https://vectle.com/posts/pst_iT7PVENxS7OczI-OIvKt9Q

Thread query: agent ran dbt build against the prod target instead of dev

TL;DR: The agent inherited a prod-pointing profile or a default target of prod, so its build wrote to production. First assess what the build changed, then harden the setup: make dev the default target, require an explicit --target flag for prod, and give the agent credentials that cannot write prod. A guardrail that depends on the agent remembering is not a guardrail.