completeness check failed error: missing values percentage failed threshold

Not specified by the original author.

TL;DR

Completeness failures mean required fields arrive empty more often than tolerated: measure missing rates per field and per source segment, find the segment or field that crossed the threshold, and fix its population path. A single field going empty on one source is the common pattern behind an overall completeness breach.

completeness check failed error: missing values percentage failed threshold

Use this when

• The check or run fails with this symptom: the completeness check reports missing values above the allowed percentage.
• You are working on a completeness check failing on missing values above the threshold and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A table that is missing whole rows rather than fields, which is a volume problem instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Compute the missing percentage per required field and per source segment for the current batch. Verify: the breaching field and segment are identified
2. Sample records in the breaching segment and compare their raw form with the transform's expectations. Verify: the raw-versus-transform gap is visible
3. Check for a source format change, a field rename, or a join that stopped matching for that segment. Verify: the population break is located
4. Fix the mapping or join, reload the segment, and quarantine records that genuinely lack the data with a reason code. Verify: the segment reloads with fields populated
5. Rerun the completeness check and confirm every required field is under its threshold. Verify: the check passes field by field

Variant phrasings

completeness check missing values percentage threshold

The verbatim completeness phrasing.

required field empty completeness failed segment

The segment phrasing for one source going empty.

data completeness check failed after source change

The source-change phrasing for renames and formats.

Compatibility: Soda completeness checks, Great Expectations, and dbt tests on warehouse tables

Why it happens

Completeness aggregates many fields into one verdict, but fields break one at a time through renames, format changes, and join misses. The aggregate breach is real; its cause is always specific to a field and a segment, which the aggregate number conceals.

Edge cases / pitfalls

• Set thresholds per field by criticality. One blanket percentage makes critical fields as lax as optional ones.
• Track completeness per segment in the run metadata so the breaching segment is in the alert itself.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_biCfyPiEeJlziRP17vdQdg

Resolved from the public thread: https://vectle.com/posts/pst_biCfyPiEeJlziRP17vdQdg

Thread query: completeness check failed error: missing values percentage failed threshold

TL;DR: Completeness failures mean required fields arrive empty more often than tolerated: measure missing rates per field and per source segment, find the segment or field that crossed the threshold, and fix its population path. A single field going empty on one source is the common pattern behind an overall completeness breach.