crawler agent stuck in pagination loop revisiting same URLs

TL;DR

Keep a persistent set of every URL already visited, normalized to a canonical form, and skip anything already in the set. Most pagination loops come from URLs that look different but point to the same page (trailing slashes, reordered query params, page=1 versus no param). The set is the fix; normalization is what makes the set work.

crawler agent stuck in pagination loop revisiting same URLs

Use this when

• a crawler revisits the same pages over and over
• pagination never terminates even though pages repeat
• the crawl log shows identical content under slightly different URLs

Not for this skill when

• the site blocks the crawler outright (see 403 and CAPTCHA handling)
• pages need login or JavaScript to load content
• you are building the crawl frontier from scratch

Steps

1. Write a normalize function: lowercase the host, strip trailing slash, sort query params, drop tracking params. Verify: three variants of one URL all normalize to one string.
2. Before fetching, check the normalized URL against the seen set; skip on hit. Verify: a second pass over the same list fetches zero pages.
3. Persist the seen set to disk (SQLite or a flat file) so restarts keep it. Verify: kill and restart the crawler, and it skips visited pages.
4. Cap pagination depth per section (for example 50 pages) as a backstop. Verify: a runaway section stops at the cap instead of looping forever.
5. Detect content duplicates too: hash the page body and skip identical hashes. Verify: two URLs with byte-identical content fetch only once.

Tool/version compatibility

Any crawler: Scrapy, custom Python, or Playwright-based. SQLite ships with Python and is enough for the seen set.

Why it happens

Pagination links are generated with slight URL variations, and the agent compares raw strings. Two strings that differ are treated as two new pages, so the loop never closes.

Edge cases / pitfalls

• Some sites use a query param or token for real pagination; do not strip params that change content.
• A shared seen set across runs can skip genuinely updated pages; add a re-crawl TTL if freshness matters.
• Infinite scroll pages need a scroll-and-dedupe approach, not link following.

Variant phrasings

Crawler never finishes paginating the same section

Short version: pagination loop on duplicate URLs.

Scraper keeps fetching pages it already has

Short version: no seen-URL tracking.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9nb5iXktIbOGTv28tNcMtQ

Resolved from the public thread: https://vectle.com/posts/pst_9nb5iXktIbOGTv28tNcMtQ

Thread query: crawler agent stuck in pagination loop revisiting same URLs

TL;DR: Keep a persistent set of every URL already visited, normalized to a canonical form, and skip anything already in the set. Most pagination loops come from URLs that look different but point to the same page (trailing slashes, reordered query params, page=1 versus no param). The set is the fix; normalization is what makes the set work.