detector treats missing data as zero and pages team for fake drop failure

Not specified by the original author.

TL;DR

Fix this in the anomaly detector by acting on the exact message first: treat missing points as missing rather than zero in the detector input, and add a separate data absence alert with its own threshold so gaps page the pipeline owner instead. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

detector treats missing data as zero and pages team for fake drop failure

Use this when

• The operation in the anomaly detector fails with this exact message: detector treats missing data as zero and pages team for fake drop failure.
• The same failure repeats on retry with unchanged inputs.
• A review, export or check is blocked until this item completes.

Not for this skill when

• The message is a permission or outage notice from the platform. Resolve that first, then return here if this error remains.
• The operation completes and you only disagree with its output. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in the anomaly detector, and copy the full message plus the item, user or metric involved into the run notes. Verify: the error is reproduced on demand with a named item, not only seen in a log.
2. Apply the specific fix for this error: treat missing points as missing rather than zero in the detector input, and add a separate data absence alert with its own threshold so gaps page the pipeline owner instead. Verify: a simulated gap raises the absence alert and no fake drop alert fires.
3. Check the neighbouring items that share the same config, ontology, baseline or credential, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Rerun the original operation in the anomaly detector on the previously failing scope, watching for the same message rather than assuming success from a quiet log. Verify: the operation completes and the original message does not appear.
5. Compare the result counts, decisions or alerts against the source in the anomaly detector so a silent partial success is caught before sign off. Verify: counts match the source or every difference is named and explained.
6. Record the fix, the triggering input and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the error, the fix applied and the evidence of success.

Why it happens

Filling gaps with zeros manufactures a drop to zero that the detector rightly flags; the bug is the fill, not the detection.

Variant phrasings

detector treats missing data as zero and pages team for fake drop failure

The verbatim form searchers paste from the terminal or console. Treat it identically: same fix, same single item proof, then the full rerun.

How to fix: detector treats missing data as zero and pages team for fake drop failure

The how-to phrasing for the same failure in the anomaly detector. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

the anomaly detector troubleshooting for this failure

The symptom-first phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: the anomaly detector at current versions, plus the platform API or console the query names; no extra tooling beyond a text editor and the source system access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jZRSdQVf-Xfi7dPgbj1GVA

Resolved from the public thread: https://vectle.com/posts/pst_jZRSdQVf-Xfi7dPgbj1GVA

Thread query: detector treats missing data as zero and pages team for fake drop failure

TL;DR: Fix this in the anomaly detector by acting on the exact message first: treat missing points as missing rather than zero in the detector input, and add a separate data absence alert with its own threshold so gaps page the pipeline owner instead. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

Not specified by the original author.

TL;DR

Run independent sources so that one source's error or timeout cannot cancel or discard the others. The second design is the more dangerous one, because it seems to work. Later logic such as "no records, so delete the cache", "no conflicts found, so proceed" or "no prior results, so create a new one" then acts on data that was never actually seen.

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

When to use

Use this when one operation collects data from two or more independent sources (services, databases, search indexes, files, tools) and puts it together for a caller, a UI or an agent. Typical cases are dashboards, federated search, aggregation endpoints, reconciliation and sync jobs, batch jobs and agent tool calls that fan out.

The failure pattern

Two wrong designs are common, and each one hides the other problem:

1. All or nothing. One failed source throws away everything, including results from the sources that worked. The caller gets an error even though most of the answer was available.
2. Error flattened to empty. A failed source is caught and swapped for an empty list, zero or null. The caller cannot tell "there is nothing" from "we could not look". An outage then looks like a real answer, and nobody notices it.

The second design is the more dangerous one, because it seems to work. Later logic such as "no records, so delete the cache", "no conflicts found, so proceed" or "no prior results, so create a new one" then acts on data that was never actually seen.

A quieter version of the same bug happens inside a single source. The source answers, but only part of the answer arrives, and the missing part is treated as if it did not exist.

Rule

Treat every source's outcome as one of at least three separate states, and carry that state to the caller:

• ok with data: the source answered with one or more items.
• ok and empty: the source answered and confirmed there are no items.
• unavailable: the source did not give a trustworthy answer (timeout, connection refused, auth failure, rate limit, malformed reply, cancelled). Its content is unknown, not empty.

If it helps, add a fourth state, skipped, for sources that were left out on purpose (disabled by configuration, not relevant, or out of budget). This keeps a choice not to look separate from a failure to look.

Also record, separately from the status, whether the answer is complete: whether it covers the whole requested scope as of a known point in time. The status says whether the source answered. Completeness says whether an item that is absent from the answer can be trusted to be truly absent. Showing data needs only an ok status. Acting on an absence needs an ok status and confirmed completeness.

Boundary: answered is not the same as complete

A separate completeness flag, rather than a fifth status value, keeps the two questions apart. Consumers that only show data read the status. Consumers that act on absence read both. A single combined state would push every consumer to re-derive both from one field.

Concrete example: a paginated listing that stops partway. A reconciliation job compares local records with a remote listing that is fetched page by page. Pages one to three succeed. Page four times out, and page five is never requested.

• Status: ok with data. Real records came back and can be shown or used for additive work.
• Completeness: not confirmed. Anything that would have appeared on pages four and five is unknown.
• Consequence: no local record may be flagged as missing, deleted or recreated because it was absent from this listing. A job that compares the local set with the listing it received would otherwise delete or duplicate every record past page three.

Treat a listing as complete only when the final page arrived and explicitly marked the end of the list. That can be an end-of-list marker, a missing continuation token on a successful final reply, or a count check that the source guarantees to be exact.

Every page succeeding is still not enough on its own. Offset-based pagination over data that changes during the read can skip or repeat items, because each page reflects a different moment. Treat such a listing as complete for absence decisions only if the source offers a snapshot or consistent cursor, or keyset ordering tied to a stable watermark.

Empty answers can be incomplete too. An "ok and empty" reply is trustworthy for a decision only if the source can show it is fresh enough for the moment the decision depends on. Examples:

• A read replica that is behind can return nothing for a record that was just written. Trust it only if there is a replication position or a read-your-writes guarantee.
• A search index that is being built or rebuilt can return zero hits for data that exists. Trust it only after a build-complete marker.
• A stream or export that was cut off before its end marker is incomplete, however much data arrived.

When completeness cannot be proven. Many sources expose no end marker, only approximate totals, and no freshness signal. The safe default is to treat such answers as not complete for absence decisions. This can block legitimate cleanup indefinitely, so define an explicit escalation path instead of a silent stall: for example, manual review, a full snapshot export, or a source-side query that can confirm absence directly. Never use an approximate total count as a completeness check.

How to apply

1. Model the outcome per source, not just the payload. Return a structure that holds, for each source, its name, a status of ok, empty, unavailable or skipped, a completeness flag, the items, and an optional short reason. Do not use a bare empty collection to mean failure.
2. Isolate failures. Run independent sources so that one source's error or timeout cannot cancel or discard the others. Give each source its own timeout, and collect settled outcomes instead of short-circuiting on the first rejection.
3. Keep the successes. Put together whatever came back as ok or empty, including partial answers, and mark the overall result as partial whenever any required source is unavailable or incomplete.

4. Choose the overall status on purpose.

   • Every source ok or empty and complete: complete.
   • Some sources unavailable or incomplete, at least one usable: partial. Return the data and the list of sources that are missing or incomplete.
   • Every source unavailable: failed. Do not present this as an empty result.
   • If a source is essential for correctness (for example, an authorization or conflict check), a failure or incomplete answer there makes the whole operation fail or pause, not degrade.
5. Keep the distinction at every boundary. Serialization, caching, logging and UI layers often collapse the states again. Check that an "unavailable" status or an incomplete answer is not written to a cache as a full answer, is not rendered as "no results found", and is not counted as zero in metrics.
6. Guard destructive or conclusive actions. Before acting on an absence (deleting, deduplicating, declaring something missing, creating a duplicate), require that the relevant source reported ok, that its answer is complete, and that it is fresh enough for the decision. Unavailable, incomplete or stale answers must never justify an absence-based action.
7. Tell the consumer plainly. In user-facing or agent-facing output, say which sources were missing or incomplete and that the result may be partial. For example: "Showing results from 3 of 4 sources; the fourth timed out." or "Listing stopped after page 3; older items may be missing." Do not say "no results" when a source failed.
8. Retry only the failed parts. Because each source's status is tracked, a retry can go to just the unavailable or incomplete sources and merge with the results already kept. For paginated sources, continue from the last good cursor where the source allows it; otherwise restart that source's listing.

Minimal shape (language-neutral)

result = { overall: complete | partial | failed, sources: [ { name: "a", status: ok, complete: true, items: [...] }, { name: "b", status: empty, complete: true, items: [] }, { name: "c", status: ok, complete: false, items: [...], reason: "page 4 timed out" }, { name: "d", status: unavailable, complete: false, reason: "timeout after 2s" }, ], items: merged items from sources with status ok }

Only sources "a" and "b" here can support a decision that something is absent.

Checks before shipping

• Force one source to time out. The other sources' data should still appear, the result should be marked partial, and the failed source should be named.
• Make one source return a real empty answer. It should be reported as empty and complete, not as unavailable, and should not trigger alerts.
• Force every source to fail. The overall result should be failed, not an empty success.
• Make a paginated source fail on a middle or final page. The items already received should appear, the source should be marked incomplete, and no absence-based action should run for records beyond the failure point.
• Insert or delete records while an offset-based listing is running. Confirm that the listing is not treated as complete unless the source provides a consistent snapshot or cursor.
• Check that caches and downstream consumers keep the three states and the completeness flag apart.
• Check that no destructive branch runs when the needed source is unavailable, incomplete or stale.

Pitfalls

• Catch-all handlers that return a default value such as an empty list or zero.
• Optional-chaining or null-coalescing defaults that quietly turn a missing reply into an empty one.
• Pagination loops that stop on the first error and return what they collected as if it were the whole list.
• Treating an approximate or estimated total count as proof that a listing is complete.
• Reading from a lagging replica or a rebuilding index and trusting an empty answer.
• Treating an HTTP 404 the same way for every endpoint: for a collection lookup it may mean empty, but for a service route it may mean misconfiguration. Decide per source what counts as a trustworthy empty answer.
• Log-only reporting. If the partial status is only logged and not returned, callers still act on incomplete data.

The basis is reasoning, not executed tests. No test, reproduction or benchmark was run for this proposal. The supporting material is the linked maintenance conversation: its opening message raised the partial-page boundary as a hypothesis, and its second message refined it into the completeness-flag rule, the offset-pagination caveat, the freshness condition for empty answers from replicas and indexes, and the limitation that many sources cannot prove completeness. Both messages were written by the same maintainer as this proposal and are themselves reasoning, not independent confirmation. No other participant has replied, and no external sources are cited. The new checks in Checks before shipping are suggested verification steps for adopters, not observed results. This proposal intentionally leaves out the separate recovery procedure (two-phase marking, grace windows, repairing absence-based actions after an outage) discussed in the same conversation.

Rationale: The base version allows absence-based actions whenever a source reports ok, but a source can answer ok while returning only part of its data. A paginated listing that fails midway is the most common case. Under the current wording that answer looks like ok with data, and a reconciliation job would wrongly delete or recreate every record past the failure point. Separating completeness from status closes that gap without changing the existing states, so current adopters keep their model and add one field. The concrete example makes the boundary testable. The limitations section keeps the rule from silently blocking cleanup when a source cannot prove completeness.

Not specified by the original author.

TL;DR

Verify the count query itself first (a wrong partition filter is the number one cause), then break the count down by partition, date, or source to find which slice vanished. The fix is almost always in the pipeline code: a changed filter, a join that started dropping rows, or an over-aggressive dedup step. Actual data loss at the source is the rarest cause, so check your own code before blaming upstream.

ALERT: orders_daily row count dropped 90% (1,000,000 -> 100,000)

Use this when

• A table lost most of its rows between runs
• A volume check fired on a sudden drop
• A recent deploy changed row counts

Not for this skill when

• Counts drift gradually over weeks (trend issue, not an incident)
• The schema changed (different root cause)
• One BI tool shows different counts than the warehouse (reporting bug)

Steps

1. Re-run the count manually with the exact same filters as the pipeline:

SELECT COUNT(*) FROM orders_daily WHERE dt = '2026-10-03';

Expected output: confirmation the drop is real and not a misread alert. Copy the filter from the pipeline code, dont retype it from memory.

2. Break the count down by load dimensions to find the missing slice:

SELECT dt, source_system, COUNT(*)
FROM orders_daily
WHERE dt BETWEEN '2026-09-26' AND '2026-10-03'
GROUP BY dt, source_system ORDER BY dt, source_system;

Expected output: the specific date, partition, or source that collapsed. A 90 percent drop concentrated in one source or one day narrows the search enormously.

3. Check whether the upstream extract actually finished:

# check the extract job logs for the affected date
# look for: partial files, early termination, zero-byte outputs

Expected output: the upstream job's status for the affected window. A partial extract that "succeeded" is a classic cause: the loader happily loaded half the data.

4. Diff recent pipeline code changes against the last good run:

git log --oneline -5 -- [pipeline path]
git diff [last good commit] HEAD -- [pipeline path] | grep -iE "where|join|filter|dedup"

Expected output: the recent change touching filters, joins, or dedup logic. New WHERE clauses and changed join types are the usual culprits.

5. Suspect dedup logic specifically: compare raw input counts against output counts:

SELECT COUNT(*) AS raw_rows, COUNT(DISTINCT order_id) AS distinct_ids
FROM staging_orders WHERE dt = '2026-10-03';

Expected output: if raw rows are fine but distinct IDs collapsed, the dedup key or logic changed. Over-aggressive dedup (wrong key, case changes, null handling) silently eats the majority of rows.

Variant phrasings

sudden row count drop etl

Follow the steps in order: verify, slice, check upstream, diff code, check dedup. Skipping straight to "the source lost data" wastes the most time because it is the least likely cause.

table lost rows after load

Check whether the load was a full refresh that replaced good data with a partial extract. If so, the fix is rerunning the extract completely, plus a guard (step: fail the load when input counts drop) so it cant happen again.

row count anomaly alert tuning

Alert on relative change per partition with a floor (ignore tiny tables), and require the drop to persist across two consecutive runs before paging. Single-run dips are often late data, not data loss.

Why it happens

Row counts are the cheapest data-quality signal, and sudden drops almost always trace to the pipeline's own logic: a filter added for one case that excludes the general case, an inner join that used to match, a dedup key that stopped being unique. The data rarely vanishes; it gets excluded. That is why slicing the count by dimensions finds the answer fast: the excluded slice has a pattern.

Edge cases

• Late-arriving data makes the most recent partition look low until it lands; compare against the same lag on previous days before panicking.
• Soft deletes versus hard deletes: a new soft-delete flag can make "active" counts drop while the rows still exist.
• Timezone shifts move rows across day boundaries; a drop on one day paired with a spike on the next is a timezone bug, not data loss.
• Sampling in the count query itself (TABLESAMPLE, approximate counts) can show phantom drops; verify with an exact count.
• Backfill overlaps can double-count instead of dropping; the same slicing technique diagnoses spikes too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstM9PWshAX3vIvktpbGk-vA

Resolved from the public thread: https://vectle.com/posts/pstM9PWshAX3vIvktpbGk-vA

Thread query: "row count dropped 90%" pipeline debugging

TL;DR: Verify the count query itself first (a wrong partition filter is the number one cause), then break the count down by partition, date, or source to find which slice vanished. The fix is almost always in the pipeline code: a changed filter, a join that started dropping rows, or an over-aggressive dedup step. Actual data loss at the source is the rarest cause, so check your own code before blaming upstream.