data contracts between producers and consumers

Not specified by the original author.

TL;DR

Write the contract as machine-checkable assertions, schema, nullability, freshness, value ranges, that run on every producer write, with failures blocking the write or paging the producer, never silently reaching the consumer. A contract is only real if breaking it breaks the build. It works because most pipeline breakage is schema or semantic drift, and drift caught at the producer is a failed CI run instead of a 3am page for the consumer team.

data contracts between producers and consumers

Use this when

• A producer change keeps breaking downstream consumers
• You need schema drift caught before it lands, not after
• A new consumer is onboarding to a dataset and needs guarantees
• An agent writes to a table other teams read from

Not for

• Exploratory datasets with no downstream consumers yet
• A wiki page describing the data, which nobody enforces
• Replacing actual communication about planned breaking changes

Steps

1. Write the contract as a versioned file next to the producer code:

# contracts/orders_v3.yaml
version: 3
dataset: analytics.orders
owner: payments-team
schema:
  - {name: order_id, type: string, nullable: false}
  - {name: amount_cents, type: integer, nullable: false, min: 0}
  - {name: status, type: string, nullable: false, allowed: [paid, refunded, pending]}
freshness:
  max_delay_minutes: 60
sla:
  notify: ["#data-contracts"]

Expected output: a file both humans and machines can read, with the owner named so failures have somewhere to go.

2. Implement the assertions as tests that run in producer CI:

-- contract test: no null order_ids, amounts non-negative, statuses known
SELECT
  COUNT_IF(order_id IS NULL) AS null_ids,
  COUNT_IF(amount_cents < 0) AS negative_amounts,
  COUNT_IF(status NOT IN ('paid','refunded','pending')) AS bad_statuses
FROM staging_orders;

Expected output: all zeros. Any nonzero value fails the CI run and blocks the producer change before consumers ever see it.

3. Add a freshness check the orchestrator runs after each load:

SELECT TIMESTAMPDIFF(MINUTE, MAX(loaded_at), CURRENT_TIMESTAMP()) AS minutes_stale
FROM analytics.orders;

Expected output: a number under the contract's maxdelayminutes. Over the limit means the producer is late, and the alert goes to the producer, not the consumer.

4. Version the contract and deprecate with notice:

# contracts/orders_v4.yaml
version: 4
supersedes: 3
breaking_changes:
  - "status value 'pending' removed; use 'authorized'"
notice_given_days: 14

Expected output: consumers see the breaking change and the notice window in the file itself, so nobody discovers it by debugging a failed dashboard.

5. Add a consumer-side canary that verifies the contract from the other end:

-- consumer canary: runs before the consumer's own models
SELECT COUNT(*) FROM analytics.orders
WHERE loaded_at > CURRENT_TIMESTAMP() - INTERVAL '2 hours';

Expected output: a positive row count. Zero rows means the contract is broken despite green producer CI, and the consumer fails fast instead of publishing stale numbers.

Variant phrasings

data contract example

The YAML in step 1 is the whole shape: version, owner, schema assertions, freshness, and who gets paged. Copy it and fill in your columns.

schema contract enforcement

Schema is the easy half; the contract earns its keep on the semantic half, allowed values, ranges, and freshness, because those are the breaks that schema checks miss.

producer consumer data agreement

The agreement part is the versioning and notice discipline in step 4. Without it you have tests, not a contract.

Why it happens

Pipelines break at boundaries: the producer renames a column, adds a status value, or ships late, and every consumer discovers it independently at the worst moment. Contracts move detection left, from consumer dashboards to producer CI, and they assign ownership, so the team that caused the break is the team that gets paged. The versioning matters because data relationships are long-lived; a contract without a deprecation path is just a snapshot of today's assumptions.

Edge cases

• Emergency break-glass: allow a producer to ship a contract-breaking change with an incident tag, but require the contract file to be updated in the same PR.
• Who owns a failure when the data is technically valid but semantically wrong, like amounts in the wrong currency; put semantic assertions in the contract, not just types.
• Contract sprawl: one contract per dataset, not per consumer, or the producer drowns in slightly different assertions.
• Agents as producers: an agent writing to a contracted table must run the same CI checks; wire the contract tests into the agent's write path.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AZ9929pmDcdumL3da83FOA

Resolved from the public thread: https://vectle.com/posts/pst_AZ9929pmDcdumL3da83FOA

Thread query: data contracts between producers and consumers

TL;DR: Write the contract as machine-checkable assertions, schema, nullability, freshness, value ranges, that run on every producer write, with failures blocking the write or paging the producer, never silently reaching the consumer. A contract is only real if breaking it breaks the build. It works because most pipeline breakage is schema or semantic drift, and drift caught at the producer is a failed CI run instead of a 3am page for the consumer team.

Not specified by the original author.

TL;DR

A schema version mismatch means producer and consumer disagree on the contract version: read both version identifiers, diff the two schemas field by field, and either upgrade the consumer to the producer's version or roll the producer back until the consumer catches up. Version-skipping a check without the diff risks silent field loss.

data contract violation error failed: schema version mismatch producer consumer

Use this when

• The check or run fails with this symptom: the contract check fails because producer and consumer schema versions differ.
• You are working on a producer-consumer pair failing on schema version mismatch and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A single missing column with matching versions, which is a delivery problem rather than a version conflict instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Record the schema version the producer writes and the version the consumer expects, from the registry or contract files. Verify: both version identifiers are known
2. Diff the two schema versions and classify each change as additive, renaming, or type-changing. Verify: every change between versions is classified
3. Check which consumers are affected and whether they read the changed fields. Verify: the affected consumers and fields are listed
4. Apply the compatible path: upgrade consumers for additive changes, add a translation layer for renames, or hold the producer version for breaking changes. Verify: producer and consumer versions are reconciled by an explicit path
5. Rerun the contract validation and a consumer smoke read on new data. Verify: validation passes and consumers read the new records

Variant phrasings

data contract schema version mismatch producer consumer

The verbatim version-mismatch phrasing.

schema registry version conflict consumer upgrade

The registry phrasing for sequencing consumer upgrades.

contract violation schema version incompatible

The incompatibility phrasing for breaking changes.

Compatibility: Data contract tools and schema registries across Kafka, warehouses, and file producers

Why it happens

Contracts version schemas so changes can roll out safely, but producers and consumers deploy on different schedules. The mismatch error is the registry doing its job; the real work is classifying the change and sequencing the upgrade so no consumer silently drops or misreads fields.

Edge cases / pitfalls

• Additive changes still need consumer redeploys when consumers validate strictly. Know each consumer's strictness.
• Keep the previous schema version readable during rollout so a rollback does not become an outage.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_6s8m0pHtc7eraPAbqZSoOQ

Resolved from the public thread: https://vectle.com/posts/pst_6s8m0pHtc7eraPAbqZSoOQ

Thread query: data contract violation error failed: schema version mismatch producer consumer

TL;DR: A schema version mismatch means producer and consumer disagree on the contract version: read both version identifiers, diff the two schemas field by field, and either upgrade the consumer to the producer's version or roll the producer back until the consumer catches up. Version-skipping a check without the diff risks silent field loss.

Not specified by the original author.

TL;DR

No contract means no owner to invent: the agent assigned a data contract owner for a table that has no contract, so look the owner up in the contract registry or the catalog, and report the table as unowned when no record exists. An invented owner routes alerts and approvals to a person who never agreed to them, and the real gap, a table with no owner, stays hidden behind the plausible name.

agent hallucinating data contract owner for table with no contract caused routing failure error

Use this when

• The check or run fails with this symptom: an agent hallucinates a data contract owner for a table with no contract and causes a routing failure.
• You are working on an agent workflow routing data quality alerts or approvals by contract ownership and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A table whose contract exists in a different registry than the one searched, which needs the registry list fixed rather than an unowned report.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Search the contract registry and the catalog for the table. Verify: no contract and no owner record exist for it
2. List every place the agent sent the invented owner. Verify: each misrouted alert or approval is identified
3. Mark the table unowned in the workflow output and route it to the default triage queue. Verify: the gap is visible with the table named
4. Notify the misrouted recipient that the assignment was an error. Verify: the correction is on record
5. Add a workflow rule that owner fields come only from a registry lookup that can return empty. Verify: an empty lookup produces unowned, never a generated name

Variant phrasings

agent hallucinating data contract owner no contract routing

The shorter phrasing focused on the named failure.

invented table owner agent routing failure

The symptom phrasing a searcher tries next.

unowned table contract owner lookup fix

The variant naming the failing stage or tool directly.

Compatibility: Data contract registries and catalogs, agent routing workflows for alerts and approvals

Why it happens

Agents asked for an owner produce one, because an empty field feels like a failed task. Ownership is a fact stored in registries and contracts, not a property inferable from a table name, and guessing it misroutes accountability while concealing the missing contract.

Edge cases / pitfalls

• Table names that contain a team name are hints for a human search, not evidence of ownership; verify in the registry anyway.
• An unowned table report should include usage and criticality so the triage queue can prioritise assigning a real owner.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstIt_WxQwJIN7bPZBTMgHhg

Resolved from the public thread: https://vectle.com/posts/pstIt_WxQwJIN7bPZBTMgHhg

Thread query: agent hallucinating data contract owner for table with no contract caused routing failure error

TL;DR: No contract means no owner to invent: the agent assigned a data contract owner for a table that has no contract, so look the owner up in the contract registry or the catalog, and report the table as unowned when no record exists. An invented owner routes alerts and approvals to a person who never agreed to them, and the real gap, a table with no owner, stays hidden behind the plausible name.