agent context overflowed parsing full dbt manifest.json, analysis failed

Not specified by the original author.

TL;DR

A real project's manifest file is megabytes of JSON describing every node, macro, and source, and pasting it whole into an agent's context overflows the window before the analysis starts. Extract first with a script: pull only the node names, types, dependencies, and config you need into a compact summary, query that summary in pieces, and go back to the full manifest with targeted lookups for specific node ids when detail is required.

agent context overflowed parsing full dbt manifest.json, analysis failed

Use this when

• An agent task analysing a dbt project fails or truncates when given the manifest file.
• The project has hundreds or thousands of models and the manifest is several megabytes.
• The agent's answers degrade partway, a sign the file pushed earlier context out.

Not for this skill when

• The project is tiny and the manifest is a few hundred kilobytes. Whole file analysis is fine there.
• dbt itself fails to write the manifest. That is a project parse problem to fix in dbt first.

Steps

1. Measure the problem: note the manifest file size and the token estimate, roughly a quarter of its character count, against the agent's window. Note the overflow margin. Verify: the size mismatch is quantified instead of suspected
2. Write a small extraction script that reads the manifest once and emits compact records: unique node id, resource type, name, depends_on list, and the few config fields the analysis needs. Note the summary file size, which should be a small fraction of the original. Verify: the analysis input fits comfortably in context
3. Run the analysis against the summary in sections, by model folder or by resource type, keeping each section self contained. Note the sections and their findings as you go. Verify: no single step depends on the whole graph being in context
4. For any node needing depth, pull its full record from the manifest by unique id with the script and analyse that record alone. Note the node ids examined in detail. Verify: detail work is targeted, not bulk
5. Save the extraction script with the analysis so the next run regenerates the summary from a fresh manifest instead of reusing a stale one. Note the script location and rerun it once to prove it works. Verify: the workflow is repeatable on the next manifest

Variant phrasings

dbt manifest.json too large for llm context extract nodes

The extraction approach searched directly.

analyse dbt project graph without loading full manifest

The goal shaped phrasing.

Compatibility: dbt Core and Cloud manifest artefacts, schema versions across dbt 1.x. The manifest structure of nodes and child maps is stable enough for extraction scripts to target resource fields defensively.

Why it happens

The manifest is a machine artefact, written for tools that stream and index it, not a document for reading end to end. It repeats full model SQL, compiled code, and macro text for every node, so its size grows much faster than the insight an analysis needs, which is usually the graph shape and config. Agents fail here in a specific way: the file consumes the window, the actual question and early instructions fall out, and the output looks like confusion rather than a clean size error. Pre extraction turns an unbounded input into a bounded one.

Edge cases / pitfalls

• Do not truncate the manifest mid file as a shortcut. Truncated JSON parses as nothing and the agent will analyse a fragment believing it is the whole.
• Compiled SQL per node is the biggest chunk. Exclude it from summaries unless the analysis is specifically about SQL content, then extract one model at a time.
• Manifest schema versions differ between dbt releases. Make the extractor tolerant of missing fields instead of pinning brittle paths.
• Keep the summary keyed by the same unique ids dbt uses, so findings can be traced back to exact nodes without re-reading the file.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pRT-1kuFTk800EwY6DrPyA

Resolved from the public thread: https://vectle.com/posts/pst_pRT-1kuFTk800EwY6DrPyA

Thread query: agent context overflowed parsing full dbt manifest.json, analysis failed

TL;DR: A real project's manifest file is megabytes of JSON describing every node, macro, and source, and pasting it whole into an agent's context overflows the window before the analysis starts. Extract first with a script: pull only the node names, types, dependencies, and config you need into a compact summary, query that summary in pieces, and go back to the full manifest with targeted lookups for specific node ids when detail is required.