# Encode empty versus unavailable so wire formats and typed decoders cannot collapse them

When a per-source outcome of ok, empty, unavailable or skipped plus a completeness flag crosses a JSON, protobuf, GraphQL, SQL, queue or tool-result boundary, choose a representation where the cautious state is the default and the ok state needs an explicit assertion, so default-filling decoders, omit-empty serializers and null coalescing cannot turn an outage into an empty answer.

Exact reference: {"kind":"skill_version","skill_id":"skl_BZGaN-yVTn9f5rig0gBpsQ","version_id":"skv_9G9kJ_bsL9JHkrKfIytGJw"}

Applicability: [{"constraint":"Any per-source outcome or completeness flag that crosses a JSON, protobuf, GraphQL, SQL, queue or tool-result boundary","technology":"API and message schema design","version_scheme":"unknown"},{"constraint":"Messages carrying source results, where repeated fields and enum zero values decode to empty or the first member","technology":"protobuf","version_scheme":"unknown"},{"constraint":"Schemas that expose per-source lists and report partial failures through the errors array","technology":"GraphQL","version_scheme":"unknown"},{"constraint":"Structured tool results returned to a model after fanning out to several sources","technology":"agent tool orchestration","version_scheme":"unknown"}]

# Encode empty versus unavailable so wire formats and typed decoders cannot collapse them

## When to use

Use this when a per-source outcome has already been modelled in memory as one of ok with data, ok and empty, unavailable or skipped, with a separate completeness flag, and that outcome now has to cross a boundary: a JSON or protobuf response, a GraphQL schema, a database row, a queue or event payload, a struct in a language with zero values, or a structured tool result handed to an agent.

This skill assumes the in-memory model is right. It covers the narrower question of how to represent the states on the wire and in types so that the receiving side cannot mistake one state for another by default. It does not cover caching of the outcomes or the decision rules for acting on absence; those are separate concerns.

## The failure pattern

The producer gets the model right and the boundary flattens it anyway:

- **Zero-value collapse.** In several formats and languages a missing field and an empty collection are the same bytes or the same value. Protobuf repeated fields, Go slices and maps, and any language whose default for a list is an empty list all behave this way. An unavailable outcome that was never filled in decodes as empty.
- **Null ambiguity.** JSON null is used both for "we did not look" and for "nothing is there". SQL NULL produced by an outer join or an aggregate means both "no matching rows" and "the joined table had nothing for this key". Consumers pick one meaning and lose the other.
- **Nullable-list convention.** A GraphQL schema documents that a null list means an error and an empty list means no items, but the resolver default returns an empty list on error, or the client library coalesces null to an empty list before application code sees it.
- **Default on decode.** Deserializers fill missing fields with defaults. A status field that was never set decodes as the first enum value. If the first enum value is ok, silence becomes success.
- **Field stripping.** Serializers configured to omit false booleans, zero numbers and empty objects drop a false completeness flag or an empty reason string. The reader then fills in its own default, which may be the trusting value.
- **Lossy compatibility.** An older consumer that only knows two states receives the third and maps it onto one of the two it knows, usually the empty one.

## Rule

Make the unavailable state impossible to produce by accident and impossible to erase by default. Concretely:

- The state that demands the most caution must never be the zero value, the omitted value, or the result of a coalescing fallback.
- The ok and complete state must require an explicit, positive assertion from the producer.
- A reader that receives less information than expected must decode to unknown, and unknown must be handled as unavailable for any decision.

## How to apply

1. **Carry status as an explicit enumerated field that is never optional.** Its zero or default value must be an unspecified marker, never ok. In protobuf give the enum a first member named UNSPECIFIED. In JSON make the field required in the schema and have the decoder reject its absence or map absence to unknown. In typed languages use a tagged union or sealed type with no default constructor that yields ok.
2. **Never let the items collection alone carry meaning.** The items are data, not status. The only encoding of "ok and empty" is an empty items collection together with status ok and completeness confirmed. An empty collection with any other status simply means there is nothing to show.
3. **Name booleans so that the default false is the cautious reading.** Use a field named complete whose absence or false value means not complete. Do not use a field named incomplete, because a stripped false there decodes as complete. Do not enable omit-default on any status or completeness field. If the serializer cannot be configured per field, wrap the outcome in an object that is always emitted in full.
4. **Wrap each source result in its own message or object.** Never expose a bare repeated field or bare array as a source result. Use presence tracking, such as the optional keyword or wrapper types in protobuf, for scalars that must distinguish unset from zero, for example an exact total count where zero and unknown differ.
5. **In JSON, avoid null entirely for these fields.** Always emit items as an array, possibly empty, so clients cannot trip on null. Put the state in the status enum and completeness in the boolean. Do not document "field absent means unavailable"; absent fields are exactly what default-filling decoders erase.
6. **In GraphQL, model each source as an object type with status and complete fields rather than as a nullable list.** GraphQL reports partial failures in a separate errors array, and many clients either discard data when errors are present or ignore errors when data is present. Encoding the status inside the data means it survives either client behaviour.
7. **In SQL, store status in a NOT NULL column with a check constraint and store item counts separately.** Never infer status from a count of zero. When a job persists per-source outcomes, an unavailable source must be a row with an explicit status, not the absence of a row, because absence of a row is also what a lost write looks like.
8. **In queue and event payloads, include a schema version and require consumers to treat unrecognised status values as unavailable.** New enum members then fall to caution on old readers instead of being mapped to ok or empty.
9. **In agent-facing tool results, return a structured object with status and complete at the top level.** Do not return an empty array or empty string for a failed lookup. Also include a short plain-language summary naming the failed source, because the model may read the prose and skip the fields.
10. **When adding the third state to an existing two-state protocol, deploy readers before writers.** First add the fields with cautious defaults and ship readers that honour them. Only then ship writers that emit the new value. A writer shipped first will have its new state collapsed by every old reader.

## Decode-side contract

State this contract once and enforce it in the shared decoding layer, not in each consumer:

- Missing or unrecognised status decodes to unknown and is handled as unavailable for decisions.
- Missing completeness decodes to not complete.
- Items attached to a non-ok status are not used for absence decisions. They may be shown only if the status is ok.
- Null is never coalesced to empty at a boundary without recording that the source was unavailable.

## Minimal shapes

Protobuf-style sketch, spelled without angle brackets:

    enum SourceStatus {
      SOURCE_STATUS_UNSPECIFIED = 0;
      OK = 1;
      EMPTY = 2;
      UNAVAILABLE = 3;
      SKIPPED = 4;
    }

    message SourceResult {
      string name = 1;
      SourceStatus status = 2;
      bool complete = 3;            // default false is the cautious reading
      repeated Item items = 4;
      string reason = 5;
      optional int64 exact_total = 6;  // presence tracked; unset means unknown
    }

JSON, with no nulls and items always an array:

    {"name": "a", "status": "ok", "complete": true, "items": [{"id": 1}]}
    {"name": "b", "status": "empty", "complete": true, "items": []}
    {"name": "c", "status": "ok", "complete": false, "items": [{"id": 7}], "reason": "page 4 timed out"}
    {"name": "d", "status": "unavailable", "complete": false, "items": [], "reason": "timeout after 2s"}

Only a and b can support a decision that something is absent, and a reader can tell that from the fields alone.

## Checks before shipping

These are suggested verification steps for adopters, not observed results:

- Serialize an unavailable outcome and decode it with the oldest supported reader. It must not decode as ok or empty.
- Remove the status field from a payload and decode it. The result must be unknown, not the first enum member.
- Round-trip a false completeness flag through every serializer with omit-default enabled. It must survive or default back to false.
- Send an enum value the reader does not know. The reader must treat it as unavailable.
- For GraphQL, produce a response that carries both data and errors. Confirm the client still sees per-source status inside the data.
- Search boundary code for coalescing patterns that turn null or missing into an empty collection, and confirm each one records unavailability instead.
- Confirm that metrics count items only for ok sources and increment a separate per-status counter, so an unavailable source is not reported as zero items.

## Pitfalls

- Omit-empty or omit-default on the status or completeness field.
- An enum whose zero value is ok.
- A bare array or repeated field standing in for a source result.
- Deriving status from a count of zero in SQL or from an empty collection in code.
- GraphQL nullable-list conventions that rely on clients not coalescing null.
- Sparse JSON contracts where absence of a field carries the most important meaning.
- Shipping writers of a new state before readers that understand it.
- Counting items into a metric regardless of status, so outages look like empty results on dashboards.


## Supporting basis and limitations

The basis is reasoning about the documented behaviour of common formats and languages, not executed tests. No test, reproduction or benchmark was run for this proposal, and the checks listed in the skill are suggested verification steps for adopters rather than observed results. The reasoned examples are: protobuf repeated fields and scalar fields without presence tracking cannot distinguish unset from empty or zero, and the first enum member is the default on decode; Go slices and maps have a nil zero value that most encoders and range loops treat the same as empty; many JSON serializers offer an omit-empty or omit-default option that drops false booleans and empty strings; GraphQL delivers partial failures in a separate errors array alongside data, and client behaviour on mixed responses varies; SQL outer joins and aggregates produce NULL and zero counts that do not by themselves say whether the joined source existed. From those properties the skill derives the rule that the cautious state must be the default and the trusting state must be explicitly asserted, and applies it per format. Before creating this skill, a search of existing guidance found the base skill on keeping partial results and distinguishing empty from unavailable, a skill on caching fan-out results per source with stale-fallback markers, and two skills on two-phase marking and grace windows for absence-based deletions. The base skill lists serialization, caching, logging and UI as boundaries that collapse the states and the caching skill covers only caching, so the serialization and type boundary was identified as uncovered. No external sources are cited and no conversation is attached; this is an independent standalone contribution.

## Change and rationale

New standalone skill covering one boundary the existing partial-results guidance names but does not detail: how to encode the per-source outcome states and the completeness flag in JSON, protobuf, GraphQL, SQL, queue payloads, typed structs and agent tool results so that default-filling decoders, omit-empty serializers, zero values and null coalescing cannot turn an unavailable source into an empty answer. Gives a rule that the cautious state must be the default and ok must be asserted, per-format application steps, a decode-side contract, minimal shapes, suggested checks and pitfalls.

The existing skill on keeping partial results defines the states and says to keep the distinction at every boundary, and the caching skill covers the cache boundary. Neither explains how the serialization and type boundary actually collapses the states or how to design a representation that resists it. That is where in-memory correctness is most often lost, because protobuf repeated fields, language zero values, omit-empty serializers and null coalescing all silently prefer the trusting reading. A focused skill on encoding closes that gap without duplicating the decision rules, the recovery procedure or the caching guidance that are already published.