Encode empty versus unavailable so wire formats and typed decoders cannot collapse them

When to use

Use this when a per-source outcome has already been modelled in memory as one of ok with data, ok and empty, unavailable or skipped, with a separate completeness flag, and that outcome now has to cross a boundary: a JSON or protobuf response, a GraphQL schema, a database row, a queue or event payload, a struct in a language with zero values, or a structured tool result handed to an agent.

This skill assumes the in-memory model is right. It covers the narrower question of how to represent the states on the wire and in types so that the receiving side cannot mistake one state for another by default. It does not cover caching of the outcomes or the decision rules for acting on absence; those are separate concerns.

The failure pattern

The producer gets the model right and the boundary flattens it anyway:

• Zero-value collapse. In several formats and languages a missing field and an empty collection are the same bytes or the same value. Protobuf repeated fields, Go slices and maps, and any language whose default for a list is an empty list all behave this way. An unavailable outcome that was never filled in decodes as empty.
• Null ambiguity. JSON null is used both for "we did not look" and for "nothing is there". SQL NULL produced by an outer join or an aggregate means both "no matching rows" and "the joined table had nothing for this key". Consumers pick one meaning and lose the other.
• Nullable-list convention. A GraphQL schema documents that a null list means an error and an empty list means no items, but the resolver default returns an empty list on error, or the client library coalesces null to an empty list before application code sees it.
• Default on decode. Deserializers fill missing fields with defaults. A status field that was never set decodes as the first enum value. If the first enum value is ok, silence becomes success.
• Field stripping. Serializers configured to omit false booleans, zero numbers and empty objects drop a false completeness flag or an empty reason string. The reader then fills in its own default, which may be the trusting value.
• Lossy compatibility. An older consumer that only knows two states receives the third and maps it onto one of the two it knows, usually the empty one.

Rule

Make the unavailable state impossible to produce by accident and impossible to erase by default. Concretely:

• The state that demands the most caution must never be the zero value, the omitted value, or the result of a coalescing fallback.
• The ok and complete state must require an explicit, positive assertion from the producer.
• A reader that receives less information than expected must decode to unknown, and unknown must be handled as unavailable for any decision.

How to apply

1. Carry status as an explicit enumerated field that is never optional. Its zero or default value must be an unspecified marker, never ok. In protobuf give the enum a first member named UNSPECIFIED. In JSON make the field required in the schema and have the decoder reject its absence or map absence to unknown. In typed languages use a tagged union or sealed type with no default constructor that yields ok.
2. Never let the items collection alone carry meaning. The items are data, not status. The only encoding of "ok and empty" is an empty items collection together with status ok and completeness confirmed. An empty collection with any other status simply means there is nothing to show.
3. Name booleans so that the default false is the cautious reading. Use a field named complete whose absence or false value means not complete. Do not use a field named incomplete, because a stripped false there decodes as complete. Do not enable omit-default on any status or completeness field. If the serializer cannot be configured per field, wrap the outcome in an object that is always emitted in full.
4. Wrap each source result in its own message or object. Never expose a bare repeated field or bare array as a source result. Use presence tracking, such as the optional keyword or wrapper types in protobuf, for scalars that must distinguish unset from zero, for example an exact total count where zero and unknown differ.
5. In JSON, avoid null entirely for these fields. Always emit items as an array, possibly empty, so clients cannot trip on null. Put the state in the status enum and completeness in the boolean. Do not document "field absent means unavailable"; absent fields are exactly what default-filling decoders erase.
6. In GraphQL, model each source as an object type with status and complete fields rather than as a nullable list. GraphQL reports partial failures in a separate errors array, and many clients either discard data when errors are present or ignore errors when data is present. Encoding the status inside the data means it survives either client behaviour.
7. In SQL, store status in a NOT NULL column with a check constraint and store item counts separately. Never infer status from a count of zero. When a job persists per-source outcomes, an unavailable source must be a row with an explicit status, not the absence of a row, because absence of a row is also what a lost write looks like.
8. In queue and event payloads, include a schema version and require consumers to treat unrecognised status values as unavailable. New enum members then fall to caution on old readers instead of being mapped to ok or empty.
9. In agent-facing tool results, return a structured object with status and complete at the top level. Do not return an empty array or empty string for a failed lookup. Also include a short plain-language summary naming the failed source, because the model may read the prose and skip the fields.
10. When adding the third state to an existing two-state protocol, deploy readers before writers. First add the fields with cautious defaults and ship readers that honour them. Only then ship writers that emit the new value. A writer shipped first will have its new state collapsed by every old reader.

Decode-side contract

State this contract once and enforce it in the shared decoding layer, not in each consumer:

• Missing or unrecognised status decodes to unknown and is handled as unavailable for decisions.
• Missing completeness decodes to not complete.
• Items attached to a non-ok status are not used for absence decisions. They may be shown only if the status is ok.
• Null is never coalesced to empty at a boundary without recording that the source was unavailable.

Minimal shapes

Protobuf-style sketch, spelled without angle brackets:

enum SourceStatus { SOURCESTATUSUNSPECIFIED = 0; OK = 1; EMPTY = 2; UNAVAILABLE = 3; SKIPPED = 4; }

message SourceResult { string name = 1; SourceStatus status = 2; bool complete = 3; // default false is the cautious reading repeated Item items = 4; string reason = 5; optional int64 exact_total = 6; // presence tracked; unset means unknown }

JSON, with no nulls and items always an array:

{"name": "a", "status": "ok", "complete": true, "items": [{"id": 1}]} {"name": "b", "status": "empty", "complete": true, "items": []} {"name": "c", "status": "ok", "complete": false, "items": [{"id": 7}], "reason": "page 4 timed out"} {"name": "d", "status": "unavailable", "complete": false, "items": [], "reason": "timeout after 2s"}

Only a and b can support a decision that something is absent, and a reader can tell that from the fields alone.

Checks before shipping

These are suggested verification steps for adopters, not observed results:

• Serialize an unavailable outcome and decode it with the oldest supported reader. It must not decode as ok or empty.
• Remove the status field from a payload and decode it. The result must be unknown, not the first enum member.
• Round-trip a false completeness flag through every serializer with omit-default enabled. It must survive or default back to false.
• Send an enum value the reader does not know. The reader must treat it as unavailable.
• For GraphQL, produce a response that carries both data and errors. Confirm the client still sees per-source status inside the data.
• Search boundary code for coalescing patterns that turn null or missing into an empty collection, and confirm each one records unavailability instead.
• Confirm that metrics count items only for ok sources and increment a separate per-status counter, so an unavailable source is not reported as zero items.

Pitfalls

• Omit-empty or omit-default on the status or completeness field.
• An enum whose zero value is ok.
• A bare array or repeated field standing in for a source result.
• Deriving status from a count of zero in SQL or from an empty collection in code.
• GraphQL nullable-list conventions that rely on clients not coalescing null.
• Sparse JSON contracts where absence of a field carries the most important meaning.
• Shipping writers of a new state before readers that understand it.
• Counting items into a metric regardless of status, so outages look like empty results on dashboards.