Encode empty versus unavailable so wire formats and typed decoders cannot collapse them

When to use

Use this when a per-source outcome has already been modelled in memory as one of ok with data, ok and empty, unavailable or skipped, with a separate completeness flag, and that outcome now has to cross a boundary: a JSON or protobuf response, a GraphQL schema, a database row, a queue or event payload, a struct in a language with zero values, or a structured tool result handed to an agent.

This skill assumes the in-memory model is right. It covers the narrower question of how to represent the states on the wire and in types so that the receiving side cannot mistake one state for another by default. It does not cover caching of the outcomes or the decision rules for acting on absence; those are separate concerns.

The failure pattern

The producer gets the model right and the boundary flattens it anyway:

• Zero-value collapse. In several formats and languages a missing field and an empty collection are the same bytes or the same value. Protobuf repeated fields, Go slices and maps, and any language whose default for a list is an empty list all behave this way. An unavailable outcome that was never filled in decodes as empty.
• Null ambiguity. JSON null is used both for "we did not look" and for "nothing is there". SQL NULL produced by an outer join or an aggregate means both "no matching rows" and "the joined table had nothing for this key". Consumers pick one meaning and lose the other.
• Nullable-list convention. A GraphQL schema documents that a null list means an error and an empty list means no items, but the resolver default returns an empty list on error, or the client library coalesces null to an empty list before application code sees it.
• Default on decode. Deserializers fill missing fields with defaults. A status field that was never set decodes as the first enum value. If the first enum value is ok, silence becomes success.
• Field stripping. Serializers configured to omit false booleans, zero numbers and empty objects drop a false completeness flag or an empty reason string. The reader then fills in its own default, which may be the trusting value.
• Lossy compatibility. An older consumer that only knows two states receives the third and maps it onto one of the two it knows, usually the empty one.
• Partial success read as full success. A source answers ok with some items but could not finish, for example because a later page timed out. The status alone looks trustworthy, and a reader that checks status without also checking completeness treats the missing remainder as genuinely absent.

Rule

Make the unavailable state impossible to produce by accident and impossible to erase by default. Concretely:

• The state that demands the most caution must never be the zero value, the omitted value, or the result of a coalescing fallback.
• The ok and complete state must require an explicit, positive assertion from the producer.
• A reader that receives less information than expected must decode to unknown, and unknown must be handled as unavailable for any decision.
• An absence decision, meaning any action taken because something was not in the result, requires both a status of ok or empty and completeness true. Status alone is never enough.

How to apply

1. Carry status as an explicit enumerated field that is never optional. Its zero or default value must be an unspecified marker, never ok. In protobuf give the enum a first member named UNSPECIFIED. In JSON make the field required in the schema and have the decoder reject its absence or map absence to unknown. In typed languages use a tagged union or sealed type with no default constructor that yields ok.
2. Never let the items collection alone carry meaning. The items are data, not status. The only encoding of "ok and empty" is an empty items collection together with status ok and completeness confirmed. An empty collection with any other status simply means there is nothing to show.
3. Name booleans so that the default false is the cautious reading. Use a field named complete whose absence or false value means not complete. Do not use a field named incomplete, because a stripped false there decodes as complete. Do not enable omit-default on any status or completeness field. If the serializer cannot be configured per field, wrap the outcome in an object that is always emitted in full.
4. Wrap each source result in its own message or object. Never expose a bare repeated field or bare array as a source result. Use presence tracking, such as the optional keyword or wrapper types in protobuf, for scalars that must distinguish unset from zero, for example an exact total count where zero and unknown differ.
5. In JSON, avoid null entirely for these fields. Always emit items as an array, possibly empty, so clients cannot trip on null. Put the state in the status enum and completeness in the boolean. Do not document "field absent means unavailable"; absent fields are exactly what default-filling decoders erase.
6. In GraphQL, model each source as an object type with status and complete fields rather than as a nullable list. GraphQL reports partial failures in a separate errors array, and many clients either discard data when errors are present or ignore errors when data is present. Encoding the status inside the data means it survives either client behaviour.
7. In SQL, store status in a NOT NULL column with a check constraint and store item counts separately. Never infer status from a count of zero. When a job persists per-source outcomes, an unavailable source must be a row with an explicit status, not the absence of a row, because absence of a row is also what a lost write looks like.
8. In queue and event payloads, include a schema version and require consumers to treat unrecognised status values as unavailable. New enum members then fall to caution on old readers instead of being mapped to ok or empty.
9. In agent-facing tool results, return a structured object with status and complete at the top level. Do not return an empty array or empty string for a failed lookup. Also include a short plain-language summary naming the failed source, because the model may read the prose and skip the fields.
10. When adding the third state to an existing two-state protocol, deploy readers before writers. First add the fields with cautious defaults and ship readers that honour them. Only then ship writers that emit the new value. A writer shipped first will have its new state collapsed by every old reader.
11. Expose one helper for absence decisions and make every consumer call it. The helper returns true only when status is ok or empty and complete is true. Consumers that need to display items may read status directly, but consumers that delete, deprovision, recreate or mark something as gone must go through the helper, so the completeness check cannot be forgotten at any one call site.

Decode-side contract

State this contract once and enforce it in the shared decoding layer, not in each consumer:

• Missing or unrecognised status decodes to unknown and is handled as unavailable for decisions.
• Missing completeness decodes to not complete.
• Items attached to a non-ok status are not used for absence decisions. They may be shown only if the status is ok.
• Items attached to an ok status with completeness false may be shown, but they are not used for absence decisions either. For the purpose of deciding that something is missing, an ok but incomplete result is handled exactly like an unavailable one.
• An absence decision is permitted only when status is ok or empty and completeness is true. Every other combination of status, completeness and items is treated as unavailable for that decision.
• Null is never coalesced to empty at a boundary without recording that the source was unavailable.

Minimal shapes

Protobuf-style sketch, spelled without angle brackets:

enum SourceStatus { SOURCESTATUSUNSPECIFIED = 0; OK = 1; EMPTY = 2; UNAVAILABLE = 3; SKIPPED = 4; }

message SourceResult { string name = 1; SourceStatus status = 2; bool complete = 3; // default false is the cautious reading repeated Item items = 4; string reason = 5; optional int64 exact_total = 6; // presence tracked; unset means unknown }

JSON, with no nulls and items always an array:

{"name": "a", "status": "ok", "complete": true, "items": [{"id": 1}]} {"name": "b", "status": "empty", "complete": true, "items": []} {"name": "c", "status": "ok", "complete": false, "items": [{"id": 7}], "reason": "page 4 timed out"} {"name": "d", "status": "unavailable", "complete": false, "items": [], "reason": "timeout after 2s"}

Only a and b can support a decision that something is absent, and a reader can tell that from the fields alone.

Worked example of the partial-success boundary

Shape c is the one most often misread, because its status is ok. Suppose a reconciliation job holds thirty local records for source c and compares them against the listing above to delete anything no longer present upstream. The upstream listing has four pages of ten items each. Pages one to three returned normally and page four timed out, so the producer correctly emitted status ok, complete false, and thirty items with one example shown.

A reader that gates deletion on status alone sees ok, finds that every local record beyond the thirty returned is missing from the listing, and deletes the ten records that live on the page that never arrived. The producer did everything right; the decision was wrong because it consulted only one of the two fields.

A reader that follows the contract sees complete false, declines to make any absence decision for source c on this run, and may still display the thirty items it did receive. Nothing is deleted, and the next run with a complete listing makes the decision correctly.

The same boundary appears whenever a source can return a prefix of its answer: cursor-based pagination that stops early, a search backend that hits a time budget and returns what it found so far, a fan-out that returns after a quorum rather than after every shard, or a stream that closed before its end marker.

Checks before shipping

These are suggested verification steps for adopters, not observed results:

• Serialize an unavailable outcome and decode it with the oldest supported reader. It must not decode as ok or empty.
• Remove the status field from a payload and decode it. The result must be unknown, not the first enum member.
• Round-trip a false completeness flag through every serializer with omit-default enabled. It must survive or default back to false.
• Send an enum value the reader does not know. The reader must treat it as unavailable.
• Feed a payload with status ok, completeness false and a non-empty items array into every code path that deletes, deprovisions, recreates or marks something as gone. Each path must decline to act. Repeat with status ok, completeness false and an empty items array, which is the shape a source produces when it timed out before its first page.
• For GraphQL, produce a response that carries both data and errors. Confirm the client still sees per-source status inside the data.
• Search boundary code for coalescing patterns that turn null or missing into an empty collection, and confirm each one records unavailability instead.
• Search consumer code for conditions that test status equal to ok without also testing complete, and confirm each one either only displays items or goes through the shared absence-decision helper.
• Confirm that metrics count items only for ok sources and increment a separate per-status counter, so an unavailable source is not reported as zero items.

Pitfalls

• Omit-empty or omit-default on the status or completeness field.
• An enum whose zero value is ok.
• A bare array or repeated field standing in for a source result.
• Deriving status from a count of zero in SQL or from an empty collection in code.
• Gating an absence decision on status ok alone, so a partial page or an early-stopped scan is read as the full set.
• GraphQL nullable-list conventions that rely on clients not coalescing null.
• Sparse JSON contracts where absence of a field carries the most important meaning.
• Shipping writers of a new state before readers that understand it.
• Counting items into a metric regardless of status, so outages look like empty results on dashboards.