# vendor portal scraper agent hung on 429 throttle loop with timeout errors: how to fix it

A vendor portal scraper hung on a 429 throttle loop needs to actually back off instead of retrying immediately. Honor the retry-after header with exponential backoff and jitter, and pause the crawl when throttled. Immediate retries turn a throttle into a ban.

```text
vendor portal scraper agent hung on 429 throttle loop with timeout errors
```

## When to use this skill
- A scraper loops on 429 responses
- Immediate retries never succeed

## When not to use this skill
- The portal blocks outright with 403 (an access issue)
- No throttle headers exist (a different failure)

## Tool compatibility
Agent web scraping with HTTP 429 handling.
Related area: browser automation and web scraping.
Cloud UIs change labels over time; if a button moved, search settings for the closest match.

## Before you start
- Sign in to the tool and keep one tab open so sessions stay clean
- Copy the exact error text and note the time it happened
- Set aside 15 to 20 minutes; most of these resolve in a single pass

## Quick diagnosis
Make sure this is the right skill before you invest the 15 minutes.
- If you are dealing with a scraper loops on 429 responses, start at step 1 below.
- If instead the portal blocks outright with 403 (an access issue), stop here; you need a different skill.
- If neither quite matches, compare your screen against the verbatim error block at the top; that block is exactly what this skill covers.

## Fix it in steps
### 1. Honor retry-after headers

```
Read the retry-after value on every 429 and wait at least that long.
```
Expected: the scraper stops hammering during the throttle window.

### 2. Add exponential backoff with jitter

```
Grow the wait between retries and randomize it.
```
Expected: retries spread out instead of clustering.

### 3. Pause the crawl when throttled

```
On repeated 429s, pause the whole crawl for several minutes.
```
Expected: the throttle clears instead of escalating.

### 4. Resume gently

```
Restart at a slower request rate than before.
```
Expected: the crawl completes without re-triggering the throttle.

## Confirm each step worked
Run through this checklist before moving on. If a check fails, redo that step instead of pushing ahead.
- Step 1 (honor retry-after headers): Expected: the scraper stops hammering during the throttle window.
- Step 2 (add exponential backoff with jitter): Expected: retries spread out instead of clustering.
- Step 3 (pause the crawl when throttled): Expected: the throttle clears instead of escalating.
- Step 4 (resume gently): Expected: the crawl completes without re-triggering the throttle.

## Quick recap
1. Honor retry-after headers
2. Add exponential backoff with jitter
3. Pause the crawl when throttled
4. Resume gently

## Still stuck?
Gather three things: the exact error text, the time it happened, and what changed just before it broke.
Check the vendor status page for an active incident. Then open a support ticket with those three things;
it cuts the back-and-forth in half. For agent runs, attach the last 50 lines of the agent log too,
since the loop or timeout signature in the log usually names the cause.

## Variant phrasings
### scraper 429 throttle loop
Back off for real, immediate retries make it worse. If this matches what you saw, the steps above apply as written.

### vendor portal throttling scraper
Slower restarts beat aggressive retries. If this matches what you saw, the steps above apply as written.

## Why it happens
Portals throttle aggressive crawlers with 429s. An agent that retries immediately treats the throttle as a transient blip, which the portal reads as continued aggression, escalating to longer throttles or blocks.

## Edge cases
- Jitter prevents synchronized retry storms
- Log every 429 with its retry-after to tune pacing
- Consider whether an API exists instead of scraping

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_F0QJ6vle8PhRzkPu0bIQaQ