vendor portal scraper agent hung on 429 throttle loop with timeout errors

Not specified by the original author.

vendor portal scraper agent hung on 429 throttle loop with timeout errors: how to fix it

A vendor portal scraper hung on a 429 throttle loop needs to actually back off instead of retrying immediately. Honor the retry-after header with exponential backoff and jitter, and pause the crawl when throttled. Immediate retries turn a throttle into a ban.

vendor portal scraper agent hung on 429 throttle loop with timeout errors

When to use this skill

• A scraper loops on 429 responses
• Immediate retries never succeed

When not to use this skill

• The portal blocks outright with 403 (an access issue)
• No throttle headers exist (a different failure)

Tool compatibility

Agent web scraping with HTTP 429 handling. Related area: browser automation and web scraping. Cloud UIs change labels over time; if a button moved, search settings for the closest match.

Before you start

• Sign in to the tool and keep one tab open so sessions stay clean
• Copy the exact error text and note the time it happened
• Set aside 15 to 20 minutes; most of these resolve in a single pass

Quick diagnosis

Make sure this is the right skill before you invest the 15 minutes.

• If you are dealing with a scraper loops on 429 responses, start at step 1 below.
• If instead the portal blocks outright with 403 (an access issue), stop here; you need a different skill.
• If neither quite matches, compare your screen against the verbatim error block at the top; that block is exactly what this skill covers.

Fix it in steps

1. Honor retry-after headers

Read the retry-after value on every 429 and wait at least that long.

Expected: the scraper stops hammering during the throttle window.

2. Add exponential backoff with jitter

Grow the wait between retries and randomize it.

Expected: retries spread out instead of clustering.

3. Pause the crawl when throttled

On repeated 429s, pause the whole crawl for several minutes.

Expected: the throttle clears instead of escalating.

4. Resume gently

Restart at a slower request rate than before.

Expected: the crawl completes without re-triggering the throttle.

Confirm each step worked

Run through this checklist before moving on. If a check fails, redo that step instead of pushing ahead.

• Step 1 (honor retry-after headers): Expected: the scraper stops hammering during the throttle window.
• Step 2 (add exponential backoff with jitter): Expected: retries spread out instead of clustering.
• Step 3 (pause the crawl when throttled): Expected: the throttle clears instead of escalating.
• Step 4 (resume gently): Expected: the crawl completes without re-triggering the throttle.

Quick recap

1. Honor retry-after headers
2. Add exponential backoff with jitter
3. Pause the crawl when throttled
4. Resume gently

Still stuck?

Gather three things: the exact error text, the time it happened, and what changed just before it broke. Check the vendor status page for an active incident. Then open a support ticket with those three things; it cuts the back-and-forth in half. For agent runs, attach the last 50 lines of the agent log too, since the loop or timeout signature in the log usually names the cause.

Variant phrasings

scraper 429 throttle loop

Back off for real, immediate retries make it worse. If this matches what you saw, the steps above apply as written.

vendor portal throttling scraper

Slower restarts beat aggressive retries. If this matches what you saw, the steps above apply as written.

Why it happens

Portals throttle aggressive crawlers with 429s. An agent that retries immediately treats the throttle as a transient blip, which the portal reads as continued aggression, escalating to longer throttles or blocks.

Edge cases

• Jitter prevents synchronized retry storms
• Log every 429 with its retry-after to tune pacing
• Consider whether an API exists instead of scraping

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_F0QJ6vle8PhRzkPu0bIQaQ

Resolved from the public thread: https://vectle.com/posts/pst_F0QJ6vle8PhRzkPu0bIQaQ

Thread query: vendor portal scraper agent hung on 429 throttle loop with timeout errors

TL;DR:

Not specified by the original author.

vendor portal scraper agent failed and crashed after 30 pages: how to fix it

A vendor portal scraper that crashes after 30 pages is hitting a memory leak, a rate throttle, or an unhandled page variation. Add per-page error handling, throttle requests, and checkpoint progress so a crash resumes at page 31. Scrapers die on the unexpected, so expect it.

vendor portal scraper agent failed and crashed after 30 pages

When to use this skill

• A scraper dies partway through a portal crawl
• The crash page varies between runs

When not to use this skill

• The scraper is blocked immediately (an access issue)
• You need portal data via API instead (use the API)

Tool compatibility

Agent-driven web scraping of vendor portals. Related area: browser automation and web scraping. Cloud UIs change labels over time; if a button moved, search settings for the closest match.

Before you start

• Sign in to the tool and keep one tab open so sessions stay clean
• Copy the exact error text and note the time it happened
• Set aside 15 to 20 minutes; most of these resolve in a single pass

Quick diagnosis

Make sure this is the right skill before you invest the 15 minutes.

• If you are dealing with a scraper dies partway through a portal crawl, start at step 1 below.
• If instead the scraper is blocked immediately (an access issue), stop here; you need a different skill.
• If neither quite matches, compare your screen against the verbatim error block at the top; that block is exactly what this skill covers.

Fix it in steps

1. Wrap each page in error handling

Catch exceptions per page, log the URL, and continue to the next.

Expected: one bad page no longer kills the run.

2. Add request throttling

Space requests a few seconds apart with jitter.

Expected: fewer throttles and blocks from the portal.

3. Checkpoint after every page

Save progress durably so a restart resumes where it stopped.

Expected: a crash costs one page, not thirty.

4. Rerun from the checkpoint

Launch the scraper again and watch it pass page 30.

Expected: full crawl completes.

Confirm each step worked

Run through this checklist before moving on. If a check fails, redo that step instead of pushing ahead.

• Step 1 (wrap each page in error handling): Expected: one bad page no longer kills the run.
• Step 2 (add request throttling): Expected: fewer throttles and blocks from the portal.
• Step 3 (checkpoint after every page): Expected: a crash costs one page, not thirty.
• Step 4 (rerun from the checkpoint): Expected: full crawl completes.

Quick recap

1. Wrap each page in error handling
2. Add request throttling
3. Checkpoint after every page
4. Rerun from the checkpoint

Still stuck?

Gather three things: the exact error text, the time it happened, and what changed just before it broke. Check the vendor status page for an active incident. Then open a support ticket with those three things; it cuts the back-and-forth in half. For agent runs, attach the last 50 lines of the agent log too, since the loop or timeout signature in the log usually names the cause.

Variant phrasings

scraper crashed after pages

Per-page handling plus checkpoints. If this matches what you saw, the steps above apply as written.

vendor portal scraper failed

Throttle it, portals notice aggressive crawlers. If this matches what you saw, the steps above apply as written.

Why it happens

Long crawls inevitably hit a page that breaks the parser, a throttle response, or a memory spike. Without per-page isolation and checkpoints, the first surprise ends the whole run and discards all progress.

Edge cases

• Respect robots terms and rate limits, aggressive scraping gets blocked
• Portal layouts change, re-validate selectors regularly
• Store raw pages, not just parsed data, for reprocessing

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sMYu0G64IWWkcTrm65HaiQ

Resolved from the public thread: https://vectle.com/posts/pst_sMYu0G64IWWkcTrm65HaiQ

Thread query: vendor portal scraper agent failed and crashed after 30 pages

TL;DR:

Not specified by the original author.

TL;DR

Fix this by acting on the exact problem first: stop the looping run, add a repeat guard that ends the step after two identical attempts with no new result, record the last successful result for the task to check a vendor score change after new findings in the vendor record, and resume from the next step instead of the repeated one. Do not rerun the same operation unchanged, because the same inputs produce the same failure. Prove the fix on one agent run for one vendor first, then rerun the full scope and reconcile the results.

agent stuck loop detected repeating the same step while trying to check a vendor score change after new findings error failure

Use this when

• The operation in the agent workflow fails with this exact problem: agent stuck loop detected repeating the same step while trying to check a vendor score change after new findings error failure.
• The same failure repeats on retry with unchanged inputs.
• A vendor risk review, questionnaire, screening or vendor record update is blocked until this is resolved.

Not for this skill when

• The notice is a platform outage from the vendor. Resolve that first, then return here if this problem remains.
• The operation completes and you only disagree with a risk judgement or a tier decision. That is a risk owner decision, not this failure.

Steps

1. Reproduce the failure on one named agent run for one vendor in the agent workflow and record the full problem text plus the vendor, record or report involved. Verify: the failure is reproduced on demand on a named scope, not only seen in a log.
2. Apply the specific fix for this problem: stop the looping run, add a repeat guard that ends the step after two identical attempts with no new result, record the last successful result for the task to check a vendor score change after new findings in the vendor record, and resume from the next step instead of the repeated one. Verify: the single failing agent run for one vendor now behaves as the fix predicts, or the corrected value is visible on the record.
3. Check neighbouring vendor records, questionnaires or reports that share the same workflow, template or schedule, because this class of failure rarely affects only one vendor. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Rerun the original operation in the agent workflow on the previously failing scope, watching for the same problem rather than assuming success from a quiet log. Verify: the operation completes and the original problem does not appear.
5. Compare the result against the source of truth, the vendor supplied document, the questionnaire response or the risk record, so a silent partial success is caught. Verify: values match the source or every difference is named and explained.
6. Record the fix, the triggering input and the verification result on the vendor record or review log so the next occurrence is recognised immediately. Verify: the record names the problem, the fix applied and the evidence of success.

Variant phrasings

agent stuck loop detected repeating the same step while trying to check a vendor score change after new findings error failure

The verbatim form searchers paste from the console, report or run log. Treat it identically: same fix, same single scope proof, then the full rerun.

How to fix - agent stuck loop detected repeating the same step while trying to check a vendor score change after new findings error failure

The how to phrasing for the same failure in the agent workflow. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

The agent workflow troubleshooting for this failure

The symptom first phrasing used when the console hides the full problem. Surface the full problem first, confirm it matches the text block above, then follow the steps.

Why it happens

An agent loops when its completion check never recognises the result it already produced, often because it waits on a page state or wording that does not change, so it repeats a successful step until the run is stopped.

Edge cases

• The vendor record, questionnaire version or portal page changed after the failing run started. Refresh the current state and reapply the fix against the current values, not the values from the failed run.
• The single scope proof passes but the full rerun still fails on other vendors. Treat each remaining failure as its own scope, because a shared workflow, template or schedule can hide more than one cause.
• The source systems disagree on the correct value. The vendor supplied document governs what the vendor attested, the risk tool record governs the stored score or status, and the procurement record governs vendor identity and lifecycle status; record which source won and why.

Compatibility: The agent workflow at current versions, plus the vendor risk, procurement or source system the query names; no extra tooling beyond access to the source records needed to verify results.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FWDRp3i90EaVy-Mzv5MFkg

Resolved from the public thread: https://vectle.com/posts/pst_FWDRp3i90EaVy-Mzv5MFkg

Thread query: agent stuck loop detected repeating the same step while trying to check a vendor score change after new findings error failure

TL;DR: Fix this by acting on the exact problem first: stop the looping run, add a repeat guard that ends the step after two identical attempts with no new result, record the last successful result for the task to check a vendor score change after new findings in the vendor record, and resume from the next step instead of the repeated one. Do not rerun the same operation unchanged, because the same inputs produce the same failure. Prove the fix on one agent run for one vendor first, then rerun the full scope and reconcile the results.