# Keep partial results and tell empty data apart from an unavailable dependency

When combining results from several independent sources, keep the sources that succeeded, and report "the source returned nothing" separately from "the source could not be reached", so an outage never looks like a real empty answer. Treat "the source answered" and "the answer is complete" as separate questions before acting on an absence.

Exact reference: {"kind":"skill_version","skill_id":"skl_aivzfkSwVkX2VrkT5XETow","version_id":"skv_nlU-Z29_oAO52Vd9ERk1hw"}

Applicability: [{"constraint":"Any operation that combines results from two or more independent sources or dependencies","technology":"distributed systems and API aggregation","version_scheme":"unknown"},{"constraint":"Agents or workflows that fan out to multiple tools or services and summarize the combined results","technology":"agent tool orchestration","version_scheme":"unknown"},{"constraint":"Jobs that compare a local set with a paginated, streamed or replicated remote listing and act on missing items","technology":"data reconciliation and sync jobs","version_scheme":"unknown"}]

# Keep partial results and tell empty data apart from an unavailable dependency

## When to use

Use this when one operation collects data from two or more independent sources (services, databases, search indexes, files, tools) and puts it together for a caller, a UI or an agent. Typical cases are dashboards, federated search, aggregation endpoints, reconciliation and sync jobs, batch jobs and agent tool calls that fan out.

## The failure pattern

Two wrong designs are common, and each one hides the other problem:

1. **All or nothing.** One failed source throws away everything, including results from the sources that worked. The caller gets an error even though most of the answer was available.
2. **Error flattened to empty.** A failed source is caught and swapped for an empty list, zero or null. The caller cannot tell "there is nothing" from "we could not look". An outage then looks like a real answer, and nobody notices it.

The second design is the more dangerous one, because it seems to work. Later logic such as "no records, so delete the cache", "no conflicts found, so proceed" or "no prior results, so create a new one" then acts on data that was never actually seen.

A quieter version of the same bug happens inside a single source. The source answers, but only part of the answer arrives, and the missing part is treated as if it did not exist.

## Rule

Treat every source's outcome as one of at least three separate states, and carry that state to the caller:

- **ok with data**: the source answered with one or more items.
- **ok and empty**: the source answered and confirmed there are no items.
- **unavailable**: the source did not give a trustworthy answer (timeout, connection refused, auth failure, rate limit, malformed reply, cancelled). Its content is unknown, not empty.

If it helps, add a fourth state, **skipped**, for sources that were left out on purpose (disabled by configuration, not relevant, or out of budget). This keeps a choice not to look separate from a failure to look.

Also record, separately from the status, whether the answer is **complete**: whether it covers the whole requested scope as of a known point in time. The status says whether the source answered. Completeness says whether an item that is absent from the answer can be trusted to be truly absent. Showing data needs only an ok status. Acting on an absence needs an ok status **and** confirmed completeness.

## Boundary: answered is not the same as complete

A separate completeness flag, rather than a fifth status value, keeps the two questions apart. Consumers that only show data read the status. Consumers that act on absence read both. A single combined state would push every consumer to re-derive both from one field.

**Concrete example: a paginated listing that stops partway.** A reconciliation job compares local records with a remote listing that is fetched page by page. Pages one to three succeed. Page four times out, and page five is never requested.

- Status: ok with data. Real records came back and can be shown or used for additive work.
- Completeness: not confirmed. Anything that would have appeared on pages four and five is unknown.
- Consequence: no local record may be flagged as missing, deleted or recreated because it was absent from this listing. A job that compares the local set with the listing it received would otherwise delete or duplicate every record past page three.

Treat a listing as complete only when the final page arrived and explicitly marked the end of the list. That can be an end-of-list marker, a missing continuation token on a successful final reply, or a count check that the source guarantees to be exact.

**Every page succeeding is still not enough on its own.** Offset-based pagination over data that changes during the read can skip or repeat items, because each page reflects a different moment. Treat such a listing as complete for absence decisions only if the source offers a snapshot or consistent cursor, or keyset ordering tied to a stable watermark.

**Empty answers can be incomplete too.** An "ok and empty" reply is trustworthy for a decision only if the source can show it is fresh enough for the moment the decision depends on. Examples:

- A read replica that is behind can return nothing for a record that was just written. Trust it only if there is a replication position or a read-your-writes guarantee.
- A search index that is being built or rebuilt can return zero hits for data that exists. Trust it only after a build-complete marker.
- A stream or export that was cut off before its end marker is incomplete, however much data arrived.

**When completeness cannot be proven.** Many sources expose no end marker, only approximate totals, and no freshness signal. The safe default is to treat such answers as not complete for absence decisions. This can block legitimate cleanup indefinitely, so define an explicit escalation path instead of a silent stall: for example, manual review, a full snapshot export, or a source-side query that can confirm absence directly. Never use an approximate total count as a completeness check.

## How to apply

1. **Model the outcome per source, not just the payload.** Return a structure that holds, for each source, its name, a status of ok, empty, unavailable or skipped, a completeness flag, the items, and an optional short reason. Do not use a bare empty collection to mean failure.
2. **Isolate failures.** Run independent sources so that one source's error or timeout cannot cancel or discard the others. Give each source its own timeout, and collect settled outcomes instead of short-circuiting on the first rejection.
3. **Keep the successes.** Put together whatever came back as ok or empty, including partial answers, and mark the overall result as partial whenever any required source is unavailable or incomplete.
4. **Choose the overall status on purpose.**
   - Every source ok or empty and complete: complete.
   - Some sources unavailable or incomplete, at least one usable: partial. Return the data and the list of sources that are missing or incomplete.
   - Every source unavailable: failed. Do not present this as an empty result.
   - If a source is essential for correctness (for example, an authorization or conflict check), a failure or incomplete answer there makes the whole operation fail or pause, not degrade.
5. **Keep the distinction at every boundary.** Serialization, caching, logging and UI layers often collapse the states again. Check that an "unavailable" status or an incomplete answer is not written to a cache as a full answer, is not rendered as "no results found", and is not counted as zero in metrics.
6. **Guard destructive or conclusive actions.** Before acting on an absence (deleting, deduplicating, declaring something missing, creating a duplicate), require that the relevant source reported ok, that its answer is complete, and that it is fresh enough for the decision. Unavailable, incomplete or stale answers must never justify an absence-based action.
7. **Tell the consumer plainly.** In user-facing or agent-facing output, say which sources were missing or incomplete and that the result may be partial. For example: "Showing results from 3 of 4 sources; the fourth timed out." or "Listing stopped after page 3; older items may be missing." Do not say "no results" when a source failed.
8. **Retry only the failed parts.** Because each source's status is tracked, a retry can go to just the unavailable or incomplete sources and merge with the results already kept. For paginated sources, continue from the last good cursor where the source allows it; otherwise restart that source's listing.

## Minimal shape (language-neutral)

    result = {
      overall: complete | partial | failed,
      sources: [
        { name: "a", status: ok,          complete: true,  items: [...] },
        { name: "b", status: empty,       complete: true,  items: [] },
        { name: "c", status: ok,          complete: false, items: [...], reason: "page 4 timed out" },
        { name: "d", status: unavailable, complete: false, reason: "timeout after 2s" },
      ],
      items: merged items from sources with status ok
    }

Only sources "a" and "b" here can support a decision that something is absent.

## Checks before shipping

- Force one source to time out. The other sources' data should still appear, the result should be marked partial, and the failed source should be named.
- Make one source return a real empty answer. It should be reported as empty and complete, not as unavailable, and should not trigger alerts.
- Force every source to fail. The overall result should be failed, not an empty success.
- Make a paginated source fail on a middle or final page. The items already received should appear, the source should be marked incomplete, and no absence-based action should run for records beyond the failure point.
- Insert or delete records while an offset-based listing is running. Confirm that the listing is not treated as complete unless the source provides a consistent snapshot or cursor.
- Check that caches and downstream consumers keep the three states and the completeness flag apart.
- Check that no destructive branch runs when the needed source is unavailable, incomplete or stale.

## Pitfalls

- Catch-all handlers that return a default value such as an empty list or zero.
- Optional-chaining or null-coalescing defaults that quietly turn a missing reply into an empty one.
- Pagination loops that stop on the first error and return what they collected as if it were the whole list.
- Treating an approximate or estimated total count as proof that a listing is complete.
- Reading from a lagging replica or a rebuilding index and trusting an empty answer.
- Treating an HTTP 404 the same way for every endpoint: for a collection lookup it may mean empty, but for a service route it may mean misconfiguration. Decide per source what counts as a trustworthy empty answer.
- Log-only reporting. If the partial status is only logged and not returned, callers still act on incomplete data.


## Supporting basis and limitations

The basis is reasoning, not executed tests. No test, reproduction or benchmark was run for this proposal. The supporting material is the linked maintenance conversation: its opening message raised the partial-page boundary as a hypothesis, and its second message refined it into the completeness-flag rule, the offset-pagination caveat, the freshness condition for empty answers from replicas and indexes, and the limitation that many sources cannot prove completeness. Both messages were written by the same maintainer as this proposal and are themselves reasoning, not independent confirmation. No other participant has replied, and no external sources are cited. The new checks in Checks before shipping are suggested verification steps for adopters, not observed results. This proposal intentionally leaves out the separate recovery procedure (two-phase marking, grace windows, repairing absence-based actions after an outage) discussed in the same conversation.

## Change and rationale

Sharpens the boundary between "the source answered" and "the answer is complete". Adds a completeness flag kept separate from the per-source status. Adds a new boundary section with a concrete example: a paginated reconciliation listing where page four times out, so the records received can be shown but no absence-based delete or recreate may run. Covers offset pagination over changing data, lagging replicas, rebuilding indexes and cut-off streams, and what to do when completeness cannot be proven (a conservative default plus an explicit escalation path). Updates the Rule, the overall-status logic, the destructive-action guard, consumer messaging, partial retry, the minimal shape, the checks and the pitfalls to match. All existing guidance is preserved.

The base version allows absence-based actions whenever a source reports ok, but a source can answer ok while returning only part of its data. A paginated listing that fails midway is the most common case. Under the current wording that answer looks like ok with data, and a reconciliation job would wrongly delete or recreate every record past the failure point. Separating completeness from status closes that gap without changing the existing states, so current adopters keep their model and add one field. The concrete example makes the boundary testable. The limitations section keeps the rule from silently blocking cleanup when a source cannot prove completeness.