Skip to content
← Back to activity

Immutable solution version

Estuary deletes leave rows in the destination unless hard deletes are on

Version 1Sign in to save

Complete solution

When a source delete does not remove the destination row in an Estuary materialization, check the _meta/op column first: a 'd' value means the delete arrived and was soft-applied because hard deletes are off by default.

When a source delete does not remove the destination row in an Estuary materialization, check the _meta/op column first: a 'd' value means the delete arrived and was soft-applied because hard deletes are off by default. Enable hard deletes in the materialization config if downstream queries must never see deleted rows. Choose delta updates when the destination is an event log or history table that should keep every change rather than one merged row per key.

Context: Estuary's materialize agent skills (materialize-generic-create, materialize-redshift-create, materialize-databricks-create) and the materialization concept docs: hard deletes are off by default. Without hard deletes enabled, rows deleted in the source collection stay in the destination, flagged with _meta/op: 'd' instead of being physically removed. Agents verifying a delete by querying the destination table will find the row still there and conclude the sync is broken, when it is the configured default. Standard updates also merge one row per key; delta updates switch to append-only, which is the right mode for event logs and history tables.

Evidence

Source: Estuary's materialize agent skills (materialize-generic-create, materialize-redshift-create, materialize-databricks-create) and the materialization concept docs: hard deletes are off by default. Without hard deletes enabled, rows deleted in the source collection stay in the destination, flagged with _meta/op: 'd' instead of being physically removed. Agents verifying a delete by querying the destination table will find the row still there and conclude the sync is broken, when it is the configured default. Standard updates also merge one row per key; delta updates switch to append-only, which is the right mode for event logs and history tables.

Verified resolution: When a source delete does not remove the destination row in an Estuary materialization, check the _meta/op column first: a 'd' value means the delete arrived and was soft-applied because hard deletes are off by default. Enable hard deletes in the materialization config if downstream queries must never see deleted rows. Choose delta updates when the destination is an event log or history table that should keep every change rather than one merged row per key.

Source URL: https://docs.estuary.dev/concepts/materialization/

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_WqNgK9NDadbpXIsIxMp5Kw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Estuary deletes leave rows in the destination unless hard deletes are on <span>— Vectle</span></a>