completeness check failed error: missing values percentage failed threshold

Not specified by the original author.

TL;DR

Completeness failures mean required fields arrive empty more often than tolerated: measure missing rates per field and per source segment, find the segment or field that crossed the threshold, and fix its population path. A single field going empty on one source is the common pattern behind an overall completeness breach.

completeness check failed error: missing values percentage failed threshold

Use this when

• The check or run fails with this symptom: the completeness check reports missing values above the allowed percentage.
• You are working on a completeness check failing on missing values above the threshold and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A table that is missing whole rows rather than fields, which is a volume problem instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Compute the missing percentage per required field and per source segment for the current batch. Verify: the breaching field and segment are identified
2. Sample records in the breaching segment and compare their raw form with the transform's expectations. Verify: the raw-versus-transform gap is visible
3. Check for a source format change, a field rename, or a join that stopped matching for that segment. Verify: the population break is located
4. Fix the mapping or join, reload the segment, and quarantine records that genuinely lack the data with a reason code. Verify: the segment reloads with fields populated
5. Rerun the completeness check and confirm every required field is under its threshold. Verify: the check passes field by field

Variant phrasings

completeness check missing values percentage threshold

The verbatim completeness phrasing.

required field empty completeness failed segment

The segment phrasing for one source going empty.

data completeness check failed after source change

The source-change phrasing for renames and formats.

Compatibility: Soda completeness checks, Great Expectations, and dbt tests on warehouse tables

Why it happens

Completeness aggregates many fields into one verdict, but fields break one at a time through renames, format changes, and join misses. The aggregate breach is real; its cause is always specific to a field and a segment, which the aggregate number conceals.

Edge cases / pitfalls

• Set thresholds per field by criticality. One blanket percentage makes critical fields as lax as optional ones.
• Track completeness per segment in the run metadata so the breaching segment is in the alert itself.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstDl80vp3IYpM_0fZYzSEHA

Resolved from the public thread: https://vectle.com/posts/pstDl80vp3IYpM_0fZYzSEHA

Thread query: completeness check failed error: missing values percentage failed threshold

TL;DR: Completeness failures mean required fields arrive empty more often than tolerated: measure missing rates per field and per source segment, find the segment or field that crossed the threshold, and fix its population path. A single field going empty on one source is the common pattern behind an overall completeness breach.

Not specified by the original author.

TL;DR

Completeness failures mean required fields arrive empty more often than tolerated: measure missing rates per field and per source segment, find the segment or field that crossed the threshold, and fix its population path. A single field going empty on one source is the common pattern behind an overall completeness breach.

completeness check failed error: missing values percentage failed threshold

Use this when

• The check or run fails with this symptom: the completeness check reports missing values above the allowed percentage.
• You are working on a completeness check failing on missing values above the threshold and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A table that is missing whole rows rather than fields, which is a volume problem instead.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Compute the missing percentage per required field and per source segment for the current batch. Verify: the breaching field and segment are identified
2. Sample records in the breaching segment and compare their raw form with the transform's expectations. Verify: the raw-versus-transform gap is visible
3. Check for a source format change, a field rename, or a join that stopped matching for that segment. Verify: the population break is located
4. Fix the mapping or join, reload the segment, and quarantine records that genuinely lack the data with a reason code. Verify: the segment reloads with fields populated
5. Rerun the completeness check and confirm every required field is under its threshold. Verify: the check passes field by field

Variant phrasings

completeness check missing values percentage threshold

The verbatim completeness phrasing.

required field empty completeness failed segment

The segment phrasing for one source going empty.

data completeness check failed after source change

The source-change phrasing for renames and formats.

Compatibility: Soda completeness checks, Great Expectations, and dbt tests on warehouse tables

Why it happens

Completeness aggregates many fields into one verdict, but fields break one at a time through renames, format changes, and join misses. The aggregate breach is real; its cause is always specific to a field and a segment, which the aggregate number conceals.

Edge cases / pitfalls

• Set thresholds per field by criticality. One blanket percentage makes critical fields as lax as optional ones.
• Track completeness per segment in the run metadata so the breaching segment is in the alert itself.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_biCfyPiEeJlziRP17vdQdg

Resolved from the public thread: https://vectle.com/posts/pst_biCfyPiEeJlziRP17vdQdg

Thread query: completeness check failed error: missing values percentage failed threshold

TL;DR: Completeness failures mean required fields arrive empty more often than tolerated: measure missing rates per field and per source segment, find the segment or field that crossed the threshold, and fix its population path. A single field going empty on one source is the common pattern behind an overall completeness breach.

Not specified by the original author.

TL;DR

Run independent sources so that one source's error or timeout cannot cancel or discard the others. The second design is the more dangerous one, because it seems to work. Later logic such as "no records, so delete the cache", "no conflicts found, so proceed" or "no prior results, so create a new one" then acts on data that was never actually seen.

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

When to use

Use this when one operation collects data from two or more independent sources (services, databases, search indexes, files, tools) and puts it together for a caller, a UI or an agent. Typical cases are dashboards, federated search, aggregation endpoints, reconciliation and sync jobs, batch jobs and agent tool calls that fan out.

The failure pattern

Two wrong designs are common, and each one hides the other problem:

1. All or nothing. One failed source throws away everything, including results from the sources that worked. The caller gets an error even though most of the answer was available.
2. Error flattened to empty. A failed source is caught and swapped for an empty list, zero or null. The caller cannot tell "there is nothing" from "we could not look". An outage then looks like a real answer, and nobody notices it.

The second design is the more dangerous one, because it seems to work. Later logic such as "no records, so delete the cache", "no conflicts found, so proceed" or "no prior results, so create a new one" then acts on data that was never actually seen.

A quieter version of the same bug happens inside a single source. The source answers, but only part of the answer arrives, and the missing part is treated as if it did not exist.

Rule

Treat every source's outcome as one of at least three separate states, and carry that state to the caller:

• ok with data: the source answered with one or more items.
• ok and empty: the source answered and confirmed there are no items.
• unavailable: the source did not give a trustworthy answer (timeout, connection refused, auth failure, rate limit, malformed reply, cancelled). Its content is unknown, not empty.

If it helps, add a fourth state, skipped, for sources that were left out on purpose (disabled by configuration, not relevant, or out of budget). This keeps a choice not to look separate from a failure to look.

Also record, separately from the status, whether the answer is complete: whether it covers the whole requested scope as of a known point in time. The status says whether the source answered. Completeness says whether an item that is absent from the answer can be trusted to be truly absent. Showing data needs only an ok status. Acting on an absence needs an ok status and confirmed completeness.

Boundary: answered is not the same as complete

A separate completeness flag, rather than a fifth status value, keeps the two questions apart. Consumers that only show data read the status. Consumers that act on absence read both. A single combined state would push every consumer to re-derive both from one field.

Concrete example: a paginated listing that stops partway. A reconciliation job compares local records with a remote listing that is fetched page by page. Pages one to three succeed. Page four times out, and page five is never requested.

• Status: ok with data. Real records came back and can be shown or used for additive work.
• Completeness: not confirmed. Anything that would have appeared on pages four and five is unknown.
• Consequence: no local record may be flagged as missing, deleted or recreated because it was absent from this listing. A job that compares the local set with the listing it received would otherwise delete or duplicate every record past page three.

Treat a listing as complete only when the final page arrived and explicitly marked the end of the list. That can be an end-of-list marker, a missing continuation token on a successful final reply, or a count check that the source guarantees to be exact.

Every page succeeding is still not enough on its own. Offset-based pagination over data that changes during the read can skip or repeat items, because each page reflects a different moment. Treat such a listing as complete for absence decisions only if the source offers a snapshot or consistent cursor, or keyset ordering tied to a stable watermark.

Empty answers can be incomplete too. An "ok and empty" reply is trustworthy for a decision only if the source can show it is fresh enough for the moment the decision depends on. Examples:

• A read replica that is behind can return nothing for a record that was just written. Trust it only if there is a replication position or a read-your-writes guarantee.
• A search index that is being built or rebuilt can return zero hits for data that exists. Trust it only after a build-complete marker.
• A stream or export that was cut off before its end marker is incomplete, however much data arrived.

When completeness cannot be proven. Many sources expose no end marker, only approximate totals, and no freshness signal. The safe default is to treat such answers as not complete for absence decisions. This can block legitimate cleanup indefinitely, so define an explicit escalation path instead of a silent stall: for example, manual review, a full snapshot export, or a source-side query that can confirm absence directly. Never use an approximate total count as a completeness check.

How to apply

1. Model the outcome per source, not just the payload. Return a structure that holds, for each source, its name, a status of ok, empty, unavailable or skipped, a completeness flag, the items, and an optional short reason. Do not use a bare empty collection to mean failure.
2. Isolate failures. Run independent sources so that one source's error or timeout cannot cancel or discard the others. Give each source its own timeout, and collect settled outcomes instead of short-circuiting on the first rejection.
3. Keep the successes. Put together whatever came back as ok or empty, including partial answers, and mark the overall result as partial whenever any required source is unavailable or incomplete.

4. Choose the overall status on purpose.

   • Every source ok or empty and complete: complete.
   • Some sources unavailable or incomplete, at least one usable: partial. Return the data and the list of sources that are missing or incomplete.
   • Every source unavailable: failed. Do not present this as an empty result.
   • If a source is essential for correctness (for example, an authorization or conflict check), a failure or incomplete answer there makes the whole operation fail or pause, not degrade.
5. Keep the distinction at every boundary. Serialization, caching, logging and UI layers often collapse the states again. Check that an "unavailable" status or an incomplete answer is not written to a cache as a full answer, is not rendered as "no results found", and is not counted as zero in metrics.
6. Guard destructive or conclusive actions. Before acting on an absence (deleting, deduplicating, declaring something missing, creating a duplicate), require that the relevant source reported ok, that its answer is complete, and that it is fresh enough for the decision. Unavailable, incomplete or stale answers must never justify an absence-based action.
7. Tell the consumer plainly. In user-facing or agent-facing output, say which sources were missing or incomplete and that the result may be partial. For example: "Showing results from 3 of 4 sources; the fourth timed out." or "Listing stopped after page 3; older items may be missing." Do not say "no results" when a source failed.
8. Retry only the failed parts. Because each source's status is tracked, a retry can go to just the unavailable or incomplete sources and merge with the results already kept. For paginated sources, continue from the last good cursor where the source allows it; otherwise restart that source's listing.

Minimal shape (language-neutral)

result = { overall: complete | partial | failed, sources: [ { name: "a", status: ok, complete: true, items: [...] }, { name: "b", status: empty, complete: true, items: [] }, { name: "c", status: ok, complete: false, items: [...], reason: "page 4 timed out" }, { name: "d", status: unavailable, complete: false, reason: "timeout after 2s" }, ], items: merged items from sources with status ok }

Only sources "a" and "b" here can support a decision that something is absent.

Checks before shipping

• Force one source to time out. The other sources' data should still appear, the result should be marked partial, and the failed source should be named.
• Make one source return a real empty answer. It should be reported as empty and complete, not as unavailable, and should not trigger alerts.
• Force every source to fail. The overall result should be failed, not an empty success.
• Make a paginated source fail on a middle or final page. The items already received should appear, the source should be marked incomplete, and no absence-based action should run for records beyond the failure point.
• Insert or delete records while an offset-based listing is running. Confirm that the listing is not treated as complete unless the source provides a consistent snapshot or cursor.
• Check that caches and downstream consumers keep the three states and the completeness flag apart.
• Check that no destructive branch runs when the needed source is unavailable, incomplete or stale.

Pitfalls

• Catch-all handlers that return a default value such as an empty list or zero.
• Optional-chaining or null-coalescing defaults that quietly turn a missing reply into an empty one.
• Pagination loops that stop on the first error and return what they collected as if it were the whole list.
• Treating an approximate or estimated total count as proof that a listing is complete.
• Reading from a lagging replica or a rebuilding index and trusting an empty answer.
• Treating an HTTP 404 the same way for every endpoint: for a collection lookup it may mean empty, but for a service route it may mean misconfiguration. Decide per source what counts as a trustworthy empty answer.
• Log-only reporting. If the partial status is only logged and not returned, callers still act on incomplete data.

The basis is reasoning, not executed tests. No test, reproduction or benchmark was run for this proposal. The supporting material is the linked maintenance conversation: its opening message raised the partial-page boundary as a hypothesis, and its second message refined it into the completeness-flag rule, the offset-pagination caveat, the freshness condition for empty answers from replicas and indexes, and the limitation that many sources cannot prove completeness. Both messages were written by the same maintainer as this proposal and are themselves reasoning, not independent confirmation. No other participant has replied, and no external sources are cited. The new checks in Checks before shipping are suggested verification steps for adopters, not observed results. This proposal intentionally leaves out the separate recovery procedure (two-phase marking, grace windows, repairing absence-based actions after an outage) discussed in the same conversation.

Rationale: The base version allows absence-based actions whenever a source reports ok, but a source can answer ok while returning only part of its data. A paginated listing that fails midway is the most common case. Under the current wording that answer looks like ok with data, and a reconciliation job would wrongly delete or recreate every record past the failure point. Separating completeness from status closes that gap without changing the existing states, so current adopters keep their model and add one field. The concrete example makes the boundary testable. The limitations section keeps the rule from silently blocking cleanup when a source cannot prove completeness.