Pinecone workflow: design metadata for filtering that actually works

# Metadata filtering: design it, do not accumulate it

## Principles

1. **Types are a contract.** `$gt` on a string field, `$in` with a scalar, `$exists` expecting a boolean: filters assume the stored type. Store every field with one consistent type across all records or filters misbehave silently.
2. **Filter on what you index.** Only metadata you actually upsert is filterable. If the query needs `tenant_id`, `doc_type`, and `published_year`, every record needs those fields.
3. **Keep filterable fields small.** Metadata caps at 40 KB per record. Bloated metadata slows writes and wastes storage; keep full document text in your own store and a pointer in metadata.
4. **Prefer equality and range on low-cardinality fields.** High-cardinality unique-per-record fields as filters are a smell; that is what ids are for.

## Recommended shape

```
{
  "tenant_id": "acme",
  "doc_type": "handbook",
  "published_year": 2025,
  "source": "support-docs",
  "chunk": 3
}
```

Flat, typed, small. Nested objects complicate filters; denormalize one level.

## Operator cheat sheet

`$eq`, `$ne` for equality; `$gt`, `$gte`, `$lt`, `$lte` for ranges on numbers; `$in`, `$nin` for lists; `$exists` for presence; `$and`, `$or` to combine. Text-match operators (`$match_phrase`, `$match_all`, `$match_any`) apply to full-text fields on document-schema indexes.

## Traps

1. Inconsistent types across records (`year` as string in old records, number in new): range filters silently drop the mismatched ones.
2. Filtering on fields that only some records have without `$exists` handling: missing field is not null, it is absent, and absent never matches `$eq`.
3. Stuffing the whole document into metadata "for convenience": hits the 40 KB cap and slows every write.