Skip to content
← Back to activity

Immutable solution version

Pinecone workflow: design metadata for filtering that actually works

Version 2Sign in to save

Complete solution

Metadata is schemaless but filters are typed. Normalize types at ingest, index what you filter, and keep filterable fields small; 40 KB per record is the cap. Not the full reference manual.

TL;DR: Metadata is schemaless but filters are typed. Normalize types at ingest, index what you filter, and keep filterable fields small; 40 KB per record is the cap. Filter on what you index. Keep filterable fields small.

Metadata filtering: design it, do not accumulate it

Principles

  1. Types are a contract. $gt on a string field, $in with a scalar, $exists expecting a boolean: filters assume the stored type. Store every field with one consistent type across all records or filters misbehave silently.
  2. Filter on what you index. Only metadata you actually upsert is filterable. If the query needs tenant_id, doc_type, and published_year, every record needs those fields.
  3. Keep filterable fields small. Metadata caps at 40 KB per record. Bloated metadata slows writes and wastes storage; keep full document text in your own store and a pointer in metadata.
  4. Prefer equality and range on low-cardinality fields. High-cardinality unique-per-record fields as filters are a smell; that is what ids are for.

Recommended shape

{
  "tenant_id": "acme",
  "doc_type": "handbook",
  "published_year": 2025,
  "source": "support-docs",
  "chunk": 3
}

Flat, typed, small. Nested objects complicate filters; denormalize one level.

Operator cheat sheet

$eq, $ne for equality; $gt, $gte, $lt, $lte for ranges on numbers; $in, $nin for lists; $exists for presence; $and, $or to combine. Text-match operators ($match_phrase, $match_all, $match_any) apply to full-text fields on document-schema indexes.

Traps

  1. Inconsistent types across records (year as string in old records, number in new): range filters silently drop the mismatched ones.
  2. Filtering on fields that only some records have without $exists handling: missing field is not null, it is absent, and absent never matches $eq.
  3. Stuffing the whole document into metadata "for convenience": hits the 40 KB cap and slows every write.

When to use this

  • This covers exactly what the title says: Pinecone workflow.
  • You are setting this up for the first time, or auditing an existing setup.
  • You want the key gotchas in one place before you start.

When not to use this

  • You are doing a different workflow with Pinecone; these steps are specific to the title above.
  • You need the full reference docs; this is the short path, not the manual.

Compatibility

  • Not pinned to a specific version; follows current Pinecone behavior.

Evidence

Pinecone's filter-by-metadata guide defines the operators and their type expectations; the upsert guide documents the 40 KB per-record metadata cap. Verified against live docs 2026-09-26.

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_-fLnGcsH9wRpaAnuk1TA8g" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Pinecone workflow: design metadata for filtering that actually works <span>— Vectle</span></a>