rate limit error hit mid-secret-sweep when agent tried checking 300 repos
Handles rate limits hitting an agent mid-secret-sweep across 300 repos: throttle, backoff, and resume. Use when sweeps die on 429s. Not for auth failures.
TL;DR
Sweeping hundreds of repos trips API rate limits. Slow the request rate, back off exponentially on 429s, and checkpoint per repo so the sweep resumes where it stopped.
Error
rate limit error hit mid-secret-sweep when agent tried checking 300 reposSteps
- Read the rate-limit headers to learn the actual budget and reset time. Expected: the real limit, not a guess.
- Throttle the agent below the limit with a fixed delay between repo checks. Expected: steady progress without 429s.
- On 429, back off exponentially and honor the reset header. Expected: the sweep pauses instead of dying.
- Checkpoint each completed repo. Expected: restarts skip finished work.
- Parallelize modestly within the budget rather than hammering. Expected: speed without bans.
When to use
- Large multi-repo sweeps hitting rate limits.
- Designing polite scan agents.
When not to use
- 401/403 errors (auth, not rate).
- A hard ban (stop and appeal, do not retry).
Tool compatibility
- GitHub, GitLab APIs; agent sweep orchestration.
Variant phrasings
429 mid sweep
Back off and resume.
rate limited scanning many repos
Throttle first.
Why it happens
APIs budget requests per token and time window; naive parallel sweeps spend the budget in seconds.
Edge cases
- Secondary rate limits trigger on patterns, not just counts; vary the pattern.
- Checkpoint per repo, not per finding, for clean resume.
- Some endpoints have separate stricter limits; the sweep must respect the tightest.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_X1ZT74YKcN2sHcrXsCJVxA