# RSS agent timed out waiting for infinite scroll results

## TL;DR
An RSS agent waiting on infinite scroll is using the wrong tool: infinite scroll is a UI pattern, and the data behind it almost always has a paginated API, a feed, or a sitemap. Do not automate the scroll; find the underlying data endpoint the page itself calls. If none exists, treat the source as unscrapeable for bulk collection and use its feed or API.

## The error
```text
(agent timeout)
rss agent timed out waiting for infinite scroll results; scroll loop never finished
```

## When this helps
- an agent times out on infinite scroll
- listing pages never finish loading
- choosing between scrolling and data endpoints
- bounding scroll automation

## When it doesn't
- the site offers a proper API; use it instead of any scrolling
- the terms forbid automated collection; skip the source
- you need the full history; scrolling cannot reliably provide it

## Works with
Any HTTP client; browser automation only as a capped last resort.

## Steps
### 1. Stop scrolling and look for the page's data API
```bash
curl -s -A "IntelBriefingBot/1.0" "https://YOUR-site/sitemap.xml" -o sitemap.xml -w "HTTP %{http_code}\n"
curl -s -A "IntelBriefingBot/1.0" "https://YOUR-site/feed" -o feed.xml -w "feed HTTP %{http_code}\n"
```
Expected: A sitemap or feed. Infinite-scroll listings are usually backed by one of these for crawlers.

### 2. Cap any scroll automation as a last resort
```python
import time
MAX_SCROLLS = 10
for i in range(MAX_SCROLLS):
    print("scroll", i + 1, "of", MAX_SCROLLS)
    time.sleep(2)
print("hard cap reached; agent moves on with what it has")
```
Expected: A bounded loop. If scrolling is ever used, it ends after 10 rounds no matter what.

### 3. Prefer the site's search or archive pages
```bash
curl -s -A "IntelBriefingBot/1.0" "https://YOUR-site/archive/2026/10" -o archive.html -w "HTTP %{http_code}\n"
grep -o "https\?://[^ \"]*" archive.html | sort -u | head -10
```
Expected: Archive page links. Date archives give the same listing content as the scroll, in one fetch.

### 4. Record the source as scroll-only if no alternative exists
```python
import json
reg = json.load(open("sources.json"))
reg["https://YOUR-site/listing"] = {"status": "scroll-only", "policy": "fetch first page only"}
json.dump(reg, open("sources.json", "w"), indent=2)
print("scroll-only source: first page only, no bulk collection")
```
Expected: A registry policy. Scroll-only sources contribute their first page; bulk history comes from feeds or archives.

## Other ways people phrase this
### infinite scroll timeout scraper
Find the data endpoint behind the scroll. The UI pattern is not a data API.

### rss agent scroll loop stuck
Cap the loop hard. Scrolling is the fallback, never the plan.

### listing page never finishes loading
Archive or sitemap pages give the same content without the scroll.

## Why it happens
Infinite scroll loads content in small batches as the user scrolls, which is hostile to automation: the agent cannot know when it is done, and each batch costs a render cycle. The page's own data API or archive serves the same items deterministically. Scrolling is the site's UI choice, not a collection interface.

## Edge cases
- Some sites lazy-load with no backing API; those are scroll-only by design.
- Scroll position affects which items load; automation gets inconsistent slices.
- The backing API may need the page's session; check the page's own requests.
- Respect robots.txt on archive and sitemap paths even when the scroll page is allowed.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fXRxabXbL-lNHzYYtKSOLw