context overflow error when agent tried to load full data quality history into context failed

TL;DR

Stop loading the full validation history into the prompt: page through stored results newest-first, keep only each suite's latest status plus counts of failing expectations, and fetch full detail only for suites that currently fail. A history summary that grows with every run will overflow any context window eventually.

context overflow error when agent tried to load full data quality history into context failed

Use this when

• The check or run fails with this symptom: the agent errors with context overflow when it loads the full data quality history.
• You are working on an agent that reports data quality status from stored validation history and the failure repeats on the same data or configuration.
• You need the failing rows, rule, or stage isolated before changing thresholds or code.

Not for this skill when

• A single validation result that is itself too large, where the fix is trimming the result payload rather than paging history.
• The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Replace the full-history read with a paged query over the results store, newest first, with a fixed page size such as 20 runs per suite. Verify: the query returns one bounded page instead of the entire history
2. Reduce each run to a compact record: suite name, run time, success flag, and failing expectation names with counts. Verify: the per-run record is a few lines, not the full result JSON
3. Stop paging once every suite has a current status or a maximum history depth is reached, and state the depth used in the report. Verify: the summary names how far back it looked per suite
4. Fetch full expectation detail only for suites whose latest run failed. Verify: detail calls happen only for failing suites

Variant phrasings

agent context overflow loading validation history

The history-loading phrasing.

context limit hit summarizing data quality results

The summarization phrasing.

agent ran out of context reading expectation results store

The results-store phrasing.

Compatibility: Agent frameworks loading validation history from Great Expectations Data Docs or a results store

Why it happens

Validation history grows without bound while the context window is fixed, and full result payloads include row samples and statistics that are enormous relative to the status information the agent actually needs. Loading everything guarantees an overflow; the failure just waits for enough history to accumulate.

Edge cases / pitfalls

• Keep a rolling per-suite status cache so repeat reports read the cache instead of re-paging history.
• If a suite has never passed, say so explicitly instead of paging back to the beginning of time.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstfKoRwn-7zw6C45NtyPHkw