schema check fails every monday after weekend backfill rewrites partitions error

TL;DR

Expect the Monday failure and design for it: weekend backfills rewrite partitions, so the schema the check validated on Friday is replaced by the backfill output shape, and the fix is aligning the backfill writer schema with the serving schema. The check is not flaky; the table genuinely has a different schema on Monday mornings. Backfill jobs often write with an older model version or a different writer that drops or reorders columns.

schema check fails every monday after weekend backfill rewrites partitions error

Use this when

• The check or run fails with this symptom: a schema check fails every Monday after a weekend backfill rewrites partitions.
• You are working on a table maintained by both a daily writer and a weekend backfill process with a schema check on top and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A one-off schema change from a release, which needs a migration rather than a backfill alignment fix.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Compare the table schema as recorded on Friday and on Monday. Verify: the exact columns that differ after the backfill are listed
2. Identify the backfill job version and the model or writer it runs. Verify: the backfill uses a different schema version than the daily writer
3. Align the backfill to the current model version and writer settings. Verify: a test backfill of one partition produces the serving schema
4. Rerun the backfill for the affected partitions. Verify: the schema check passes on the rewritten partitions
5. Add a schema assertion to the backfill job itself so it fails before writing a mismatched partition. Verify: the backfill refuses a wrong-shape write in a test run

Variant phrasings

schema check fails monday weekend backfill partitions

The shorter phrasing focused on the named failure.

backfill rewrites partitions schema mismatch error

The symptom phrasing a searcher tries next.

weekend backfill breaks schema check fix

The variant naming the failing stage or tool directly.

Compatibility: dbt and Spark backfill workflows, partitioned warehouse and lakehouse tables

Why it happens

Backfills are often pinned to an older code version for reproducibility, or run through a separate writer with its own schema handling. Every weekend they overwrite good partitions with the old shape, and the schema check correctly reports the table it finds on Monday.

Edge cases / pitfalls

• Pinning the backfill to old code is sometimes deliberate for value reproducibility; align the schema handling even when the value logic stays pinned.
• Partition-level schema differences can survive a table-level fix; verify per partition after the rerun, not only the table schema.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_OAQoSpbJkxgetelTb-rSiw

Resolved from the public thread: https://vectle.com/posts/pst_OAQoSpbJkxgetelTb-rSiw

Thread query: schema check fails every monday after weekend backfill rewrites partitions error

TL;DR: Expect the Monday failure and design for it: weekend backfills rewrite partitions, so the schema the check validated on Friday is replaced by the backfill output shape, and the fix is aligning the backfill writer schema with the serving schema. The check is not flaky; the table genuinely has a different schema on Monday mornings. Backfill jobs often write with an older model version or a different writer that drops or reorders columns.