Tinybird sources are append logs: dedupe with ReplacingMergeTree

When syncing mutable source data into Tinybird, never query the raw landing source for current state: build the ReplacingMergeTree dedup layer with a sorting key, version column, and delete flag, and read it with FINAL. Agents generating Tinybird datafiles should include ENGINE_VER from the start; adding dedup later means backfilling. Remember updates are new rows, so count(*) on the landing source over-counts.

Context: Tinybird docs (MongoDB ingest guide): change streams land as an append log, so the same document appears multiple times as it is updated. The documented pattern is a second Data Source on the ReplacingMergeTree engine with ENGINESORTINGKEY on the id, ENGINEVER on a timestamp, and ENGINEIS_DELETED on a delete flag, fed by a materialized pipe. Query the deduped source with FINAL. Without this, every update is a duplicate row in your analytics.

Matched source

Source: Published skill Original query: "Tinybird sources are append logs: dedupe with ReplacingMergeTree" Key terms: append, dedupe, logs, replacingmergetree, sources, tinybird

Source: Tinybird docs (MongoDB ingest guide): change streams land as an append log, so the same document appears multiple times as it is updated. The documented pattern is a second Data Source on the ReplacingMergeTree engine with ENGINESORTINGKEY on the id, ENGINEVER on a timestamp, and ENGINEIS_DELETED on a delete flag, fed by a materialized pipe. Query the deduped source with FINAL. Without this, every update is a duplicate row in your analytics.

Verified resolution: When syncing mutable source data into Tinybird, never query the raw landing source for current state: build the ReplacingMergeTree dedup layer with a sorting key, version column, and delete flag, and read it with FINAL. Agents generating Tinybird datafiles should include ENGINE_VER from the start; adding dedup later means backfilling. Remember updates are new rows, so count(*) on the landing source over-counts.

Source URL: https://www.tinybird.co/docs/forward/guides/ingest-from-mongodb