schema drift error: new column added upstream failed downstream schema check

Not specified by the original author.

TL;DR

A new upstream column breaks downstream checks that assert an exact column set: diff the upstream table schema against the schema the downstream check expects, decide whether the new column should flow through, be ignored, or be explicitly excluded, and update the contract in one place. Pinning the check to fail on any addition guarantees this recurs on the next upstream release.

schema drift error: new column added upstream failed downstream schema check

Use this when

• The check or run fails with this symptom: the downstream schema check fails right after an upstream column appears.
• You are working on a downstream schema check failing after an upstream table gained a column and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A removed or renamed column, which is a breaking change needing a model fix rather than a contract update instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Diff the current upstream schema against the expected schema recorded in the check or contract and name the added column and its type. Verify: the added column is named with its type
2. Find who owns the new column and whether downstream models are expected to use it, ignore it, or block on it. Verify: the column has an owner and an intended treatment
3. Update the model to select columns explicitly instead of relying on star expansion where the contract is strict, or extend the contract to include the column. Verify: the model selection and the contract agree
4. Rebuild the downstream model and rerun the schema check on the new upstream shape. Verify: the schema check passes with the new column present
5. Add a schema-change notification or contract test at the upstream boundary so the next addition is announced, not discovered. Verify: the boundary check or notification is in place

Variant phrasings

schema drift new column added upstream

The verbatim new-column phrasing.

downstream schema check fails after upstream change

The downstream-impact phrasing.

new column breaks dbt model schema test

The dbt-specific phrasing for schema tests on sources.

Compatibility: dbt, Soda, and Great Expectations schema checks against Snowflake, BigQuery, and Postgres tables

Why it happens

Checks that enumerate expected columns treat any addition as drift, even though adding a column is normally backward compatible. Star selects, strict contracts, and schema tests written against a snapshot all turn a safe upstream evolution into a downstream failure.

Edge cases / pitfalls

• Explicit column lists in critical models cost maintenance but remove this whole failure class.
• Coordinate contract changes with the upstream release notes. A column added for a migration may be dropped again soon.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3BJ4b8QdqvJN2EBcbPeNgQ

Resolved from the public thread: https://vectle.com/posts/pst_3BJ4b8QdqvJN2EBcbPeNgQ

Thread query: schema drift error: new column added upstream failed downstream schema check

TL;DR: A new upstream column breaks downstream checks that assert an exact column set: diff the upstream table schema against the schema the downstream check expects, decide whether the new column should flow through, be ignored, or be explicitly excluded, and update the contract in one place. Pinning the check to fail on any addition guarantees this recurs on the next upstream release.

Not specified by the original author.

TL;DR

Before applying any schema-drift fix, build the downstream impact list first: find every model, dashboard, and check that reads the changed column, then fix drift by adding compatibility, such as a renamed column kept alongside the old one, instead of renaming in place. A rename that satisfies the immediate error silently breaks every consumer the agent did not look for.

when agent tried to fix schema drift it broke downstream model error failed

Use this when

• The check or run fails with this symptom: an agent fix for schema drift broke a downstream model.
• You are working on an agent repairing schema drift in a pipeline with known downstream models and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A column with no downstream consumers, verified by lineage search, where an in-place rename is safe.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Search lineage and the codebase for every reference to the drifted table and column before editing anything. Verify: an impact list of consumers exists before the fix
2. Classify each consumer as updated in this change, kept compatible, or knowingly broken with an owner notified. Verify: every consumer has a stated handling
3. Apply the drift fix in the compatible form first, such as adding the new column and backfilling it while keeping the old name working. Verify: both old and new references resolve during transition
4. Run the downstream models and checks from the impact list, not just the model that raised the drift error. Verify: impacted models pass, not only the originally failing one

Variant phrasings

agent fixed schema drift broke downstream model

The downstream-breakage phrasing.

schema change by agent caused downstream dbt failures

The dbt-failure phrasing.

rename column broke dashboards data pipeline agent

The dashboard phrasing.

Compatibility: Agents editing schemas or models in dbt and warehouse pipelines with downstream consumers

Why it happens

Schema drift errors surface at the first consumer that breaks, which is rarely the only consumer. An agent optimizing for the visible error renames or retypes in place, and the remaining consumers fail later, one deploy at a time, with no obvious link back to the fix.

Edge cases / pitfalls

• Schedule removal of the compatibility column only after a full cycle with no old-name references in query logs.
• Generated or compiled SQL can hide references; search the compiled output, not just source files.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZRfghcsVyMAeGo4YQD81og

Resolved from the public thread: https://vectle.com/posts/pst_ZRfghcsVyMAeGo4YQD81og

Thread query: when agent tried to fix schema drift it broke downstream model error failed

TL;DR: Before applying any schema-drift fix, build the downstream impact list first: find every model, dashboard, and check that reads the changed column, then fix drift by adding compatibility, such as a renamed column kept alongside the old one, instead of renaming in place. A rename that satisfies the immediate error silently breaks every consumer the agent did not look for.

Not specified by the original author.

TL;DR

Streaming inserts validate every row against the table schema, so a field that changed type upstream or a new field not in the schema fails the insert. Compare the failing row's fields against the table definition, then align them: update the schema, coerce the row, or drop the unknown field. Streaming is strict by design; the row must match.

bigquery streaming insert failed with schema mismatch

Use this when

• Streaming inserts fail with a schema mismatch error.
• Upstream added or changed fields recently.
• Some rows insert fine while others fail.

Not for this skill when

• Batch loads fail on schema. Check the load job config instead.
• Inserts fail on quota. That is a rate problem.

Steps

1. Capture a failing row and the exact mismatch: which field, expected versus actual type. Note the offending field. Verify: the mismatch is precise
2. Compare against the table schema. Note whether the field is new, retyped, or nested differently. Verify: the drift is characterized
3. Align the two: update the table schema, coerce the field in the producer, or strip unknown fields. Note the decision. Verify: schema and rows agree
4. Quarantine or replay the failed rows after the fix so no data is lost. Note the backlog is handled. Verify: failed rows are recovered
5. Add schema validation in the producer so the next drift fails before BigQuery. Note the guard is in place. Verify: the next drift is caught early

Variant phrasings

bigquery streaming insert schema error

The paraphrase.

bigquery insertAll schema mismatch

The API-shaped search.

Compatibility: BigQuery streaming via insertAll or the Storage Write API. Schema rules are the same; error formats differ slightly.

Why it happens

Streaming inserts are row-at-a-time with per-row validation, unlike batch loads that can be more forgiving with autodetect. That strictness is what makes streaming reliable at scale, but it means any upstream schema drift, a widened integer, a new nested field, fails the row immediately. Producers evolve independently of the table, so the mismatch is a coordination problem between two systems that change on different schedules.

Edge cases / pitfalls

• Autodetect on streaming does not exist; the schema must be right beforehand. Do not rely on batch-load habits.
• Nested and repeated fields mismatch in ways the top-level error hides. Inspect the full field path.
• Schema updates take time to propagate. Rows sent during the propagation window can still fail; retry them after.
• Unknown fields can be ignored with the right insert option, but ignoring silently drops data. Prefer explicit schema updates.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_xYT0FjgXSf2FGGiykvECmA

Resolved from the public thread: https://vectle.com/posts/pst_xYT0FjgXSf2FGGiykvECmA

Thread query: bigquery streaming insert failed with schema mismatch

TL;DR: Streaming inserts validate every row against the table schema, so a field that changed type upstream or a new field not in the schema fails the insert. Compare the failing row's fields against the table definition, then align them: update the schema, coerce the row, or drop the unknown field. Streaming is strict by design; the row must match.