## TL;DR

Sweeping hundreds of repos trips API rate limits. Slow the request rate, back off exponentially on 429s, and checkpoint per repo so the sweep resumes where it stopped.

## Error

```text
rate limit error hit mid-secret-sweep when agent tried checking 300 repos
```

## Steps

1. Read the rate-limit headers to learn the actual budget and reset time. Expected: the real limit, not a guess.
2. Throttle the agent below the limit with a fixed delay between repo checks. Expected: steady progress without 429s.
3. On 429, back off exponentially and honor the reset header. Expected: the sweep pauses instead of dying.
4. Checkpoint each completed repo. Expected: restarts skip finished work.
5. Parallelize modestly within the budget rather than hammering. Expected: speed without bans.

## When to use

- Large multi-repo sweeps hitting rate limits.
- Designing polite scan agents.

## When not to use

- 401/403 errors (auth, not rate).
- A hard ban (stop and appeal, do not retry).

## Tool compatibility

- GitHub, GitLab APIs; agent sweep orchestration.

## Variant phrasings

### 429 mid sweep

Back off and resume.

### rate limited scanning many repos

Throttle first.

## Why it happens

APIs budget requests per token and time window; naive parallel sweeps spend the budget in seconds.

## Edge cases

- Secondary rate limits trigger on patterns, not just counts; vary the pattern.
- Checkpoint per repo, not per finding, for clean resume.
- Some endpoints have separate stricter limits; the sweep must respect the tightest.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_X1ZT74YKcN2sHcrXsCJVxA