Shape partial aggregate responses so a naive client cannot read an unavailable source as empty

When to use

Use this when a service, endpoint or agent tool combines several independent sources and returns the combined answer to consumers you do not control: other teams, generated SDK clients, mobile apps, spreadsheets, or an agent harness that reads tool output. It assumes the server already models each source outcome as ok, empty, unavailable or skipped with a completeness flag, and keeps the sources that succeeded. It covers the next narrower question: how to lay out the response so that the distinction survives a client that never looks at the status field.

The failure pattern

The server does everything right internally and still ships an outage as an empty answer, because the wire shape lets the client collapse the states again:

1. Empty collection for a failed source. The per-source entry carries status unavailable, but its items field is an empty list because the serializer requires the field. A client that reads only items sees nothing and renders "no results".
2. Default-to-empty deserializers. A generated client marks the items field as required and fills a missing value with an empty list. Even when the server omits the field, the client materializes an empty answer.
3. Partial marked only deep inside. The overall result is flagged partial only within per-source entries. A client that reads the merged top-level list has no single field to check.
4. Wrong transport status. A partial answer is sent as a 5xx. Retry logic and circuit breakers discard the good data and retry everything, so the healthy sources are hit again during the outage. Or the opposite: a total failure is sent as a 200 with an empty merged list.
5. Retrofit onto a bare-list endpoint. A status field is added to an endpoint that used to return a plain list. Old clients ignore the new field and keep treating partial as complete.
6. Truncated source shipped as whole. A paged upstream returns its first page and then times out. The entry is serialized as ok with a plausible items list and nothing beside it says the list stopped early. A client reading only items treats the truncated list as the complete answer, which is the same flattening bug one level down: not unknown read as empty, but partial read as whole.

Rule

Make the naive reading of the response fail safe. A consumer that reads only the data fields must end up with "unknown", an error or an obviously missing value, never with a confirmed empty answer, whenever a source was unavailable or incomplete. A consumer that reads the items of a truncated source must not be able to mistake them for the whole answer without opting in. Carry the overall status and the list of missing sources at the top level of the body. Use the transport status only to separate usable from unusable responses.

The truncated-source boundary

A source outcome sits in one of four places, and the boundary between the middle two is where most contracts are vague:

• ok and complete. The source answered fully. Items present, may be non-empty.
• ok and not complete. The source answered in part: a first page arrived and later pages did not, a streaming read was cut off, or the source itself reported that it stopped early. Items present and real, but the list is a prefix of an unknown whole.
• empty. The source answered fully and had nothing. Items present and empty. This is the only state that carries an empty list.
• unavailable or skipped. No answer. No items field at all.

The rule for the second state: keep status ok, set the per-source completeness flag false, carry the items that did arrive, set the top-level completeness flag false, and add the source to the missing list with reason incomplete. Do not promote a truncated source to unavailable, because that throws away real items that a consumer displaying a list can still use. Do not leave it as plain ok, because then absence from its list looks like proof.

Concrete case. A forum source is paged at fifty items per page. The aggregator receives page one with fifty items and the request for page two times out. The wrong serialization is status ok, completeness true, fifty items: a support macro that searches the list for an existing thread does not find it on page one, concludes it does not exist, and opens a duplicate. The right serialization is status ok, completeness false, fifty items, top-level completeness false, missing list naming the forum with reason incomplete. The macro checks the top-level flag, sees false, and skips creation. A client that only renders the list still shows the fifty items and a "more may exist" marker because the accessor returned them with a partial marker rather than as a bare list.

How to apply

1. Never serialize an empty collection for an unavailable or skipped source. Omit the items field or send null. Reserve an empty list for the empty status only. If the schema language supports variants, model the per-source outcome as a tagged union where only the ok and empty variants carry an items field at all, and where the ok variant carries the completeness flag as a required field with no default.
2. Serialize a truncated source as ok with completeness false. Carry its items. Never emit a truncated list under completeness true, and never drop the items by relabeling the source unavailable. Add the source to the top-level missing list with reason incomplete so a consumer that reads only the missing list still learns that this source is not a basis for absence decisions.
3. Put one overall status at the top level. Include overall as complete, partial or failed, a boolean complete flag, and a missing list naming each source that was unavailable, incomplete or skipped with a short classified reason such as timeout, auth, rate_limit, malformed, incomplete or disabled. Do not send raw error strings; they leak host names, paths and internal identifiers.
4. Keep the merged list honest. If a merged items list is offered for convenience, document that it contains only items from ok sources, that items from a truncated source are included, and that absence from it proves nothing unless complete is true. Consider not offering a merged list at all for endpoints where consumers make absence-based decisions.
5. Choose the transport status by usability, not by perfection. Complete and partial responses are 2xx with the body-level status carrying the detail. A failed response, meaning no usable source or an essential source unavailable, is a 5xx with no data body that could be mistaken for content. A truncated essential source is partial, not failed, because it carries usable items. Reasons: retry and breaker logic keys on the transport status, and a partial 5xx throws away good data and amplifies load on the failing source. A 200 with an empty merged list for total failure is the flattened-error bug at the transport layer.
6. Declare which sources are essential. A source is essential when the endpoint's consumers make decisions on that source's absence. Declare the essential set per endpoint in the contract rather than inferring it per request from which sources happened to succeed. Concrete case: a product page combines a pricing source and a reviews source. Pricing is down, reviews are healthy. Consumers decide whether an item is purchasable on the presence of a price, so the response is failed and 5xx even though one source succeeded. The reverse case, reviews down and pricing healthy, is partial and 2xx with reviews in the missing list. One healthy secondary source does not make a response usable.
7. Add a degraded marker outside the body. Set a response header or equivalent metadata that names the missing sources, so proxies, logs and intermediate caches can see degradation without parsing the body. Give partial responses a private or no-store cache directive so an intermediate cache does not store them with the normal lifetime.
8. Fix the client side of the contract. In generated clients and hand-written parsers, declare the items field optional or nullable with no default. Provide one accessor that returns the items for ok-and-complete and empty outcomes, returns the items together with an explicit partial marker for ok-and-not-complete, and raises or returns an explicit unknown value for the others. A caller that wants a bare list from a truncated source must opt in by name. Reject code that reads a nullable items field with an or-empty default.
9. Do not retrofit partial onto a bare-list contract. If an existing endpoint returns a plain list, old clients cannot see any new status field. Either version the endpoint and return the structured shape only to callers that request the new version, or keep the old endpoint all-or-nothing and return a 5xx when any source fails or is truncated. Never start returning a shorter plain list to old clients.
10. For agent tool output, say it first. When the consumer is a language model reading text, place a one-line degradation notice at the very start of the text output, naming the missing and truncated sources, and also include the structured fields. A notice appended after a long list is easily dropped from a summary.
11. Keep the shape identical across success and degradation. Do not switch to a different top-level shape when degraded. Clients written against the success shape must parse the degraded one and find the status where they expect it.

Minimal shape (language-neutral)

response body = { overall: complete or partial or failed, complete: true or false, missing: [ { source, reason: timeout or auth or rate_limit or malformed or incomplete or disabled } ], sources: [ { name: "a", status: ok, complete: true, items: [...] }, { name: "b", status: empty, complete: true, items: [] }, { name: "c", status: unavailable, complete: false } no items field at all { name: "d", status: ok, complete: false, items: [...] } truncated: real items, listed in missing with reason incomplete ], items: merged items from ok sources only, including the truncated ones }

Transport: 2xx for complete and partial, 5xx for failed, plus a degraded header on partial. Failed means no usable source or a declared essential source unavailable; a truncated essential source is still partial.

Reasoned example

A search aggregator combines a product index, a help-center index and a community forum. The forum service goes down. The server correctly marks the forum unavailable, but the serializer always emits items as an empty list, and the mobile client was generated from a schema where items is required. The mobile app shows a search with product and help results and a forum section that says "no discussions yet". A support macro that reads the forum count decides to open a new thread because none exist, and creates duplicates all afternoon.

With the shape above, the forum entry has no items field, the client accessor returns unknown for it, the app renders "forum results unavailable", the top-level complete flag is false, and the macro checks that flag and skips thread creation. The response is still a 200, so the app does not retry the product and help indexes, and a degraded header lets the edge cache bypass storage.

The same aggregator on a different afternoon: the forum is up but slow, and only its first page arrives before the deadline. The forum entry is status ok, completeness false, with the first page of items, and the forum appears in the missing list with reason incomplete. The app renders the first page with a "more may exist" marker, the macro sees the top-level flag false and skips creation, and the response is again a 200 with a degraded header. Had the server relabeled the slow forum as unavailable, the app would have shown nothing from it; had it left the entry as plain ok, the macro would have created duplicates again.

These examples are reasoning about the procedure. They are not the result of an executed test.

Checks before shipping

These are suggested verification steps for adopters, not observed results.

• Serialize a response with one unavailable source and inspect the raw bytes. The unavailable entry must have no items field or a null one, never an empty list.
• Serialize a response with one truncated source and inspect the raw bytes. The entry must have status ok, completeness false and its received items, and the top-level missing list must name it with reason incomplete.
• Generate or compile the client from the published schema and read items for the unavailable source without checking status. The read must fail or return an explicit unknown, not an empty collection.
• Read items for the truncated source through the same accessor. The result must carry a partial marker, and obtaining a bare list must require an explicit opt-in.
• Force every source to fail. The transport status must be 5xx and the body must not contain a merged items list.
• Force only a declared essential source to fail while a secondary source succeeds. The transport status must be 5xx.
• Force one non-essential source to fail. The transport status must be 2xx, the top-level complete flag false, and the missing list must name that source with a classified reason and no raw error text.
• Send a partial response through any intermediate cache or proxy and confirm the degraded marker survives and the response is not stored as fresh.
• Call the endpoint from an old client that predates the status field, with one source failing or truncated, and confirm the old client receives an error rather than a shorter list.

Pitfalls

• Schema tools that make every array field required and default it to empty.
• Serializing a truncated source as ok with completeness true, or dropping its items by relabeling it unavailable.
• A completeness flag with a default of true in the schema, so an omitted flag reads as complete.
• Sending partial as 5xx, which discards the good sources and multiplies retries during an outage.
• Sending total failure as 200 with an empty list.
• Treating a response as usable because some source succeeded, when the source consumers actually decide on is the one that failed.
• Marking partial only inside per-source entries and offering a top-level merged list with no flag beside it.
• Raw exception messages in per-source reasons.
• A different top-level shape for degraded responses that clients written against the success shape cannot parse.
• Adding a status field to a plain-list endpoint and assuming old clients will honor it.
• Tool output for an agent that mentions degradation only at the end of a long listing.