agent re-scraped the same billing page in a loop, export failed

TL;DR

The scraper's next-page detection is broken: it never sees the end. Log every URL fetched, stop when a URL repeats, fix the next-page selector, and cap total pages.

agent re-scraped the same billing page in a loop, export failed

Use this when

• A billing-page scrape fetches the same URL repeatedly.
• The crawl never terminates.
• The export fails because the crawl never finishes.

Not for this skill when

• A single page times out. That is a wait problem.
• The pages are distinct but numerous. That is a large crawl, not a loop.
• The site blocks the crawler. Use the sanctioned path.

Steps

1. Log every URL the scraper fetches. Verify: the log shows the repetition.
2. Stop the crawl when a URL repeats. Verify: the loop breaks.
3. Fix the next-page selector or the end-of-list detection. Verify: the last page is recognized.
4. Cap total pages as a backstop. Verify: the cap is enforced.
5. Confirm the crawl ends with complete data. Verify: the export succeeds.

Variant phrasings

scraper loop

The short phrasing.

same page scraped

The behavior-first search.

crawl never ends

The symptom-first phrasing.

Compatibility: Any browser automation crawling paginated billing pages.

Why it happens

Single-page apps often keep the next button visible on the last page, or the next-page token stops advancing. The scraper must detect repeated content itself.

Edge cases / pitfalls

• Infinite-scroll pages: cap by item count, not page count.
• Session expiry mid-crawl redirecting to login, which looks like new pages; detect the redirect.
• Query parameters that change cosmetically between fetches; normalize URLs before deduping.
• The fix being in the wait: sometimes the next page loads slowly and the scraper re-reads the current one.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstG4K5xjtYG7epO5eSmixNg

Resolved from the public thread: https://vectle.com/posts/pstG4K5xjtYG7epO5eSmixNg

Thread query: agent re-scraped the same billing page in a loop, export failed

TL;DR: The scraper's next-page detection is broken: it never sees the end. Log every URL fetched, stop when a URL repeats, fix the next-page selector, and cap total pages.