## TL;DR
Shrink what the agent validates in memory: run Pandera in a chunked pass over the data, validate dtypes and schema once on a representative chunk, then apply row checks per chunk and aggregate the failure cases. Loading a huge frame plus full failure-case collection in one agent step is what blows the timeout and the memory together.

```text
agent timed out when agent tried to run pandera validation on large dataframe
```

## Use this when
- The check or run fails with this symptom: the agent's Pandera run on the large frame exceeds its timeout.
- You are working on an agent timing out running Pandera validation on a large dataframe and the failure repeats on the same data or configuration.
- You need the failing rows, rule, or stage isolated before changing thresholds or code.

## Not for this skill when
- Validating a small frame that fits comfortably in memory, where chunking adds needless complexity instead.
- The error text names a different tool, stage, or root cause. Match the verbatim error first and use that skill.

## Steps
1. Measure the frame's size and the single-pass validation time on a ten percent slice to project the full cost. Verify: the projected full cost is measured from a slice
2. Split validation into a schema and dtype pass on one chunk, then row-level checks applied chunk by chunk. Verify: validation is restructured into passes
3. Cap failure-case collection per check and store counts plus samples instead of every failing row. Verify: failure reporting is capped with counts preserved
4. Stream chunks from the source rather than materialising the entire frame in the agent's memory. Verify: the agent processes streamed chunks
5. Run the full chunked validation and confirm it finishes with complete counts and sampled failure cases. Verify: the run completes with full coverage counts

## Variant phrasings
### agent timed out pandera validation large dataframe
The verbatim Pandera-timeout phrasing.
### pandera chunked validation large dataframe agent
The chunking phrasing for pass-based validation.
### pandera failure cases cap counts samples large frame
The failure-capping phrasing for large frames.

Compatibility: Pandera validation run by an agent on large Pandas or Polars DataFrames

## Why it happens
Pandera validates the frame it is given, all at once, and collects rich failure detail. An agent that materialises a very large frame and asks for full detail pays the memory and time cost together, while chunked validation keeps the same logical coverage at a fraction of the peak cost.

## Edge cases / pitfalls
- Chunk boundaries must not split groups that checks evaluate together, such as uniqueness across the whole frame; handle those with a separate global pass.
- Record the chunking rule in the run so failure counts are interpretable later.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VnIC4S_e-QLO4MSP1XrJmw