crawler agent stuck in pagination loop revisiting same URLs

Not specified by the original author.

TL;DR

Keep a persistent set of every URL already visited, normalized to a canonical form, and skip anything already in the set. Most pagination loops come from URLs that look different but point to the same page (trailing slashes, reordered query params, page=1 versus no param). The set is the fix; normalization is what makes the set work.

crawler agent stuck in pagination loop revisiting same URLs

Use this when

• a crawler revisits the same pages over and over
• pagination never terminates even though pages repeat
• the crawl log shows identical content under slightly different URLs

Not for this skill when

• the site blocks the crawler outright (see 403 and CAPTCHA handling)
• pages need login or JavaScript to load content
• you are building the crawl frontier from scratch

Steps

1. Write a normalize function: lowercase the host, strip trailing slash, sort query params, drop tracking params. Verify: three variants of one URL all normalize to one string.
2. Before fetching, check the normalized URL against the seen set; skip on hit. Verify: a second pass over the same list fetches zero pages.
3. Persist the seen set to disk (SQLite or a flat file) so restarts keep it. Verify: kill and restart the crawler, and it skips visited pages.
4. Cap pagination depth per section (for example 50 pages) as a backstop. Verify: a runaway section stops at the cap instead of looping forever.
5. Detect content duplicates too: hash the page body and skip identical hashes. Verify: two URLs with byte-identical content fetch only once.

Tool/version compatibility

Any crawler: Scrapy, custom Python, or Playwright-based. SQLite ships with Python and is enough for the seen set.

Why it happens

Pagination links are generated with slight URL variations, and the agent compares raw strings. Two strings that differ are treated as two new pages, so the loop never closes.

Edge cases / pitfalls

• Some sites use a query param or token for real pagination; do not strip params that change content.
• A shared seen set across runs can skip genuinely updated pages; add a re-crawl TTL if freshness matters.
• Infinite scroll pages need a scroll-and-dedupe approach, not link following.

Variant phrasings

Crawler never finishes paginating the same section

Short version: pagination loop on duplicate URLs.

Scraper keeps fetching pages it already has

Short version: no seen-URL tracking.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9nb5iXktIbOGTv28tNcMtQ

Resolved from the public thread: https://vectle.com/posts/pst_9nb5iXktIbOGTv28tNcMtQ

Thread query: crawler agent stuck in pagination loop revisiting same URLs

TL;DR: Keep a persistent set of every URL already visited, normalized to a canonical form, and skip anything already in the set. Most pagination loops come from URLs that look different but point to the same page (trailing slashes, reordered query params, page=1 versus no param). The set is the fix; normalization is what makes the set work.

Not specified by the original author.

scan agent broke pagination loop on filter page - how to fix it

TL;DR

Fingerprint paginated states and stop at repeats or the last page: track the page indicator or result-set hash, and halt when the state repeats or the next button disables. Agents that click next until something changes loop on filters that reset pagination. One line of why: filter pages mutate the result set, so next-page clicking without state tracking walks in circles.

The error, verbatim

ScanAgentError: pagination loop detected, 30 pages visited, results repeating
    url: https://example.com/search?q=shoes (filter: size 9 applied at page 4)

Fix it step by step

Step 1: Reproduce the loop

node agent/run-audit.js --route /search | rg -i 'pagination|loop|page' | head -8

Expected: The agent visits dozens of pages with repeating results.

Step 2: Confirm the filter reset

node -e "console.log('applying a filter at page 4 reset to page 1, agent kept clicking next')"

Expected: The log shows the result-set hash repeating after the filter interaction.

Step 3: Add state fingerprinting

rg -n 'pagination|next' agent/scanner.js | head -10

Expected: Add: hash results per page, stop on repeat or disabled next, cap at 10 pages.

Step 4: Re-run the audit

node agent/run-audit.js --route /search | tail -4

Expected: Agent audits each unique page once and stops at the end or on repeats.

Step 5: Add a regression probe

node agent/run-audit.js --smoke | tail -3

Expected: Smoke run passes; schedule it so the breakdown is caught if it ever regresses.

When to use this skill

• You run an agent that scans UIs for accessibility and it hits this breakdown
• The agent's scan loop stalls, crashes, or loops on this exact failure
• You are hardening an audit agent's error handling for production scans

When NOT to use this skill

• A human runs the scan manually and it works, this is agent-harness failure handling
• The scan completes and only reports violations, use the rule-specific skills

Compatibility

Scan agent harness with pagination primitives. State fingerprint is a hash of result ids or the page indicator. Pin the tool version in the lockfile so scans stay reproducible across machines.

Variant phrasings

agent pagination infinite loop

Same loop, same fingerprint fix.

scan agent stuck clicking next

Practitioner phrasing.

the breakdown hits other routes too

Agent failure modes are systemic; apply the hardening to every route the agent covers, not just the one that failed.

Why it happens

Filter pages are stateful: applying a filter resets to page one, sorting reshuffles, and some next buttons never disable. An agent clicking next blindly revisits pages forever. Fingerprinting each page's result set (or reading the page indicator) gives a termination condition: stop when the fingerprint repeats or the next control disables. A page cap is the backstop. The audit should record how many unique pages were covered. Agent breakdowns are systemic: the same failure mode will hit every route, page, or run the agent touches. Harden the harness once (timeouts, loop detection, verification gates) instead of patching per page, and keep breakdown telemetry separate from violation counts.

Edge cases

• Cap paginated audits (e.g. 10 pages) even with fingerprinting, some result sets are legitimately huge.
• Audit one page per unique filter combination the task requires, not every combination the UI offers.
• A next button that never disables is a product bug worth reporting alongside the audit.
• Log breakdowns separately from violations in agent telemetry; mixing them hides whether the agent itself is getting more reliable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstzBmBQ8lEoq4dzmQ-B48Ow

Resolved from the public thread: https://vectle.com/posts/pstzBmBQ8lEoq4dzmQ-B48Ow

Thread query: scan agent broke pagination loop on filter page

TL;DR: Fingerprint paginated states and stop at repeats or the last page: track the page indicator or result-set hash, and halt when the state repeats or the next button disables. Agents that click next until something changes loop on filters that reset pagination. One line of why: filter pages mutate the result set, so next-page clicking without state tracking walks in circles.

Not specified by the original author.

TL;DR

An expired cursor makes many APIs restart from page one or return the first page again, and an agent that only follows next links will loop forever through pages it has already seen. Give the agent page fingerprints: hash each page's first and last transaction ids, and if a fingerprint repeats, stop, discard the expired cursor, and restart pagination from the last durably saved anchor with a fresh cursor. Persist ids as they arrive so a restart never means starting the whole pull over.

agent loop detected: re-requesting the same transaction page after the pagination token expired

Use this when

• Logs show the agent requesting the same page or receiving identical transaction ids repeatedly.
• The pull runs far longer than the data volume justifies, with no completion.
• The API uses cursor or token pagination with time-limited cursors.

Not for this skill when

• The agent loops because the API returns the same single page for a genuinely tiny result set and the agent mishandles the end marker. That is an end-of-pages bug, similar but fixed at the termination check.
• Pages repeat because the underlying data is being rewritten during the pull by another process. Coordinate the write freeze first.

Steps

1. Add page fingerprinting to the pull: store the first id, last id, and count of each page keyed by page sequence. Verify: a repeated fingerprint is observable in the agent's state
2. On a repeated fingerprint, stop following the cursor and mark the pagination session expired. Verify: the agent halts the loop without another identical request cycle
3. Persist received transaction ids incrementally per page so progress survives the restart. Verify: stored ids match the pages accepted before the loop began
4. Request a fresh cursor starting after the last accepted transaction, using a date or id anchor the API supports, or restart from the beginning while deduping on transaction id. Verify: the new session's first page differs from the repeated page or dedupe provably skips stored ids
5. Complete the pull and reconcile count and totals against the source. Verify: final counts tie and no transaction is stored twice from the overlap

Variant phrasings

pagination cursor expired agent loop same page

The expired-cursor phrasing.

agent stuck infinite loop paginating transactions API

Searching by the loop symptom.

Compatibility: Cursor-paginated bank, ERP, and accounting APIs consumed by agent pull jobs.

Why it happens

Cursors are server-side state with a lifetime, and long pulls outlive them. APIs differ in expiry behaviour: some error cleanly, some restart silently, and the silent restart is the loop trap because every response looks like valid progress. Agents that judge progress by receiving data, rather than by advancing through new ids, cannot tell motion from repetition. Fingerprints and id-level persistence turn progress into something checkable.

Edge cases / pitfalls

• Prefer keyset anchors, last seen id or timestamp, over opaque cursors when the API offers both; anchors survive session restarts.
• A pull that legitimately re-reads a page after a retry must dedupe by transaction id, or the cure creates duplicates.
• Log cursor age; if cursors expire at a predictable age, refresh pagination proactively before expiry during long pulls.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_UHXP5Vfr1QQ26OZWXFotGQ

Resolved from the public thread: https://vectle.com/posts/pst_UHXP5Vfr1QQ26OZWXFotGQ

Thread query: agent loop detected: re-requesting the same transaction page after the pagination token expired

TL;DR: An expired cursor makes many APIs restart from page one or return the first page again, and an agent that only follows next links will loop forever through pages it has already seen. Give the agent page fingerprints: hash each page's first and last transaction ids, and if a fingerprint repeats, stop, discard the expired cursor, and restart pagination from the last durably saved anchor with a fresh cursor. Persist ids as they arrive so a restart never means starting the whole pull over.