Greenhouse 429 "too many requests" on the Harvest API - the agent's bulk export has no pagination backoff

Greenhouse 429 "too many requests" on the Harvest API - the agent's bulk export has no pagination backoff

TL;DR

The bulk export paged as fast as it could and hit the Harvest rate limit. Pace the pages, honor the retry-after header on 429, and checkpoint progress so a throttled run resumes instead of restarting. Bulk reads are guests; act like one.

429 Too Many Requests (Greenhouse Harvest API bulk export)

Steps

1. Read the rate-limit headers on the 429: limit, remaining, and retry-after. Expected: You know the allowed pace.

2. Add pacing between pages: a fixed delay or token bucket sized under the limit. Expected: Requests stay under the ceiling.

3. Honor retry-after on 429: sleep the requested duration, then continue. Expected: Throttles pause instead of failing.

4. Checkpoint after each page so a killed run resumes where it stopped. Expected: Restarts do not re-fetch everything.

5. Schedule bulk exports off-peak and consider the webhook feed for incremental updates instead of full exports. Expected: The export stops being a daily hammer.

When to use

• Harvest bulk reads hit 429
• exports have no pacing
• the export restarts from scratch on failure

When not to use

• the 429 is on writes (different limits, different pacing)
• a single request 429s (check for a loop, not a limit)
• the limit is genuinely too low for the business need (talk to Greenhouse)

Tool compatibility

• Greenhouse Harvest API
• bulk export workflows

Variant phrasings

• "greenhouse 429 too many requests"
• "harvest api rate limit export"
• "greenhouse bulk export throttled"

Why it happens

The Harvest API rate-limits like any shared API, and an unpaced export looks like an attack: hundreds of pages as fast as the loop runs. The 429 is the server asking for manners. Pacing plus retry-after plus checkpoints turns the export from a hammer into a guest.

Edge cases

• the retry-after value is authoritative; do not substitute your own backoff
• checkpoints must be durable; in-memory progress dies with the process
• incremental webhook feeds beat full exports for freshness and politeness

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gbxYSguJhouWF3SamPGtFA