Immutable solution version
Pinecone workflow: design metadata for filtering that actually works
Complete solution
Metadata is schemaless but filters are typed. Normalize types at ingest, index what you filter, and keep filterable fields small; 40 KB per record is the cap.
Metadata filtering: design it, do not accumulate it
Principles
- Types are a contract.
$gton a string field,$inwith a scalar,$existsexpecting a boolean: filters assume the stored type. Store every field with one consistent type across all records or filters misbehave silently. - Filter on what you index. Only metadata you actually upsert is filterable. If the query needs
tenant_id,doc_type, andpublished_year, every record needs those fields. - Keep filterable fields small. Metadata caps at 40 KB per record. Bloated metadata slows writes and wastes storage; keep full document text in your own store and a pointer in metadata.
- Prefer equality and range on low-cardinality fields. High-cardinality unique-per-record fields as filters are a smell; that is what ids are for.
Recommended shape
{
"tenant_id": "acme",
"doc_type": "handbook",
"published_year": 2025,
"source": "support-docs",
"chunk": 3
}Flat, typed, small. Nested objects complicate filters; denormalize one level.
Operator cheat sheet
$eq, $ne for equality; $gt, $gte, $lt, $lte for ranges on numbers; $in, $nin for lists; $exists for presence; $and, $or to combine. Text-match operators ($match_phrase, $match_all, $match_any) apply to full-text fields on document-schema indexes.
Traps
- Inconsistent types across records (
yearas string in old records, number in new): range filters silently drop the mismatched ones. - Filtering on fields that only some records have without
$existshandling: missing field is not null, it is absent, and absent never matches$eq. - Stuffing the whole document into metadata "for convenience": hits the 40 KB cap and slows every write.
Evidence
Pinecone's filter-by-metadata guide defines the operators and their type expectations; the upsert guide documents the 40 KB per-record metadata cap. Verified against live docs 2026-09-26.
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_-fLnGcsH9wRpaAnuk1TA8g" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Pinecone workflow: design metadata for filtering that actually works <span>— Vectle</span></a>