## TL;DR

Profiling tables one API call at a time trips warehouse rate limits. The agent should pull metadata in bulk from the information schema, process tables in chunks with pauses between them, and checkpoint after each chunk so a limit hit resumes instead of restarting.

## Error

```text
rate limit error hit mid-warehouse scan when agent tried profiling 400 tables
```

## Steps

1. Replace per-table API calls with bulk information schema queries that return metadata for many tables at once. Expected: one query replaces hundreds of calls.
2. Split the remaining per-table work into chunks (for example 25 tables per chunk). Expected: each chunk is a bounded unit of work.
3. Pause briefly between chunks. Expected: the request rate stays under the limit.
4. Checkpoint completed chunks to a scratch file after each one. Expected: a rate-limit hit resumes from the last checkpoint.
5. On a 429, back off exponentially before continuing. Expected: the scan completes without hammering the warehouse.

## When to use

- An agent profiling hundreds of tables hits rate limits.
- Any bulk warehouse metadata scan.

## When not to use

- Profiling a handful of tables (limits are unlikely).
- The limit is on the dbt Cloud API rather than the warehouse (different throttle).

## Tool compatibility

- All warehouses with information schema access; dbt's adapter connection or direct SQL.

## Variant phrasings

### 429 during schema crawl

Same fix: bulk queries, chunks, checkpoints, backoff.

### Profiling slowdown misread as a hang

Rate-limit backoff looks like slowness; log the 429s so it is visible.

## Why it happens

Warehouses throttle metadata API calls per second. An agent firing one call per table for 400 tables exceeds the budget in seconds, and the warehouse starts rejecting requests.

## Edge cases

- Some warehouses throttle by concurrency, not just rate; limit parallel workers too.
- Caching metadata for the session avoids re-profiling tables the agent already saw.
- Sampling (profiling a subset) is legitimate when the goal is a representative picture, not a census.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_61YKFxbe_OfP3fwSETiYBw