data dictionary as agent context: how to maintain one

TL;DR

Keep the dictionary as code, YAML next to the models, with the boring parts (types, nullability, sample values) auto-generated from the schema and the meaning parts written by humans during PR review. Serve the agent only the slice relevant to its task, not the whole dictionary. It works because a dictionary rots when updating it is a separate chore, so generation removes the chore and PR review enforces the meaning, while slicing keeps the agent's context focused.

data dictionary as agent context: how to maintain one

Use this when

• Agents query tables whose columns they keep misreading
• Column semantics are tribal knowledge, like what status really means
• You are onboarding humans too, not just agents
• An agent needs to pick the right table among similar ones

Not for

• A one-time documentation dump nobody updates
• Undocumented exploratory tables with no owner
• Replacing a conversation with the table owner about intent

Steps

1. Define the entry schema once, keep it small:

# dictionaries/orders.yaml
table: analytics.orders
owner: payments-team
columns:
  - name: status
    type: string
    nullable: false
    meaning: "order lifecycle state; 'pending' means authorized not captured"
    sample_values: [paid, refunded, pending]
  - name: amount_cents
    type: integer
    nullable: false
    meaning: "charged amount in cents, always positive; refunds are separate rows"

Expected output: a file with one entry per column that matters. The meaning field is the part the agent actually needs; everything else is context.

1. Auto-generate the boring parts from the warehouse catalog:

-- nightly job refreshes types, nullability, and sample values
SELECT column_name, data_type, is_nullable
FROM information_schema.columns
WHERE table_schema = 'analytics' AND table_name = 'orders';

Expected output: a generated diff against the YAML. New columns appear as entries missing a meaning, which becomes a task, not a mystery.

1. Require human-written meaning in PR review for new columns:

PR CHECKLIST - dictionary:
[ ] every new column has a meaning entry
[ ] sample_values reflect current data, not launch-day data
[ ] owner field names a team, not a person

Expected output: no column ships without its meaning documented. The checklist is short on purpose; long checklists get skipped.

1. Serve the agent the relevant slice, retrieved by task:

# retrieve dictionary entries for tables the agent's query touches
relevant = dictionary.search(query_tables=["analytics.orders", "analytics.customers"])
context = "\n".join(f"{c['name']}: {c['meaning']}" for c in relevant.columns)

Expected output: a compact block of column meanings injected into the agent's context. The agent sees dozens of lines, not thousands.

1. Check dictionary freshness in CI so it cannot silently rot:

-- fail CI if any column lacks a meaning entry
SELECT table_name, column_name FROM dictionary_coverage
WHERE meaning IS NULL;

Expected output: zero rows on a healthy repo. Any row is a specific, assignable task: write the meaning for this column.

Variant phrasings

data dictionary for LLM context

The LLM half is the slicing in step 4. Full-dictionary dumps waste context and bury the relevant definitions under noise.

maintain data catalog for agents

Maintenance is the generation plus the PR checklist. A catalog without a maintenance loop is a snapshot with an expiry date.

column descriptions best practice

Describe meaning and gotchas, not types. The warehouse already knows the type; only a human knows that pending means authorized-not-captured.

Why it happens

Agents hallucinate semantics for columns they do not understand: status becomes whatever the model guesses, and amount_cents gets divided by 100 twice. The dictionary fixes this with ground truth written by the people who own the data. It rots when documentation is a separate task from shipping, which is why the meaning lives in the same PR as the column and the boring parts are generated. Slicing matters because an agent with the full dictionary in context will still skim; an agent with twelve relevant lines will read them.

Edge cases

• PII in descriptions or sample values: keep sample values generic and never put real customer data in the dictionary.
• Multiple sources of truth: if dbt docs, a wiki, and the YAML disagree, pick one canonical file and generate the others from it.
• Rapidly evolving tables: the freshness check will complain constantly; that is the check working, assign the meaning-writing, do not disable the check.
• Agents may still misread ambiguous meanings; when a misread happens, fix the meaning text, the dictionary is the patch surface.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst0LBq77ZP2y89BRXrHqXlQ