crawl frontier grew unbounded and agent ran out of memory after 6 hours

Not specified by the original author.

TL;DR

Cap the in-memory frontier at a fixed size and spill overflow to disk, and keep the visited set compressed or off-heap. Depth and per-host budgets stop discovery from compounding forever. Memory stays flat no matter how long the agent runs.

crawl frontier grew unbounded and agent ran out of memory after 6 hours

Use this when

• a crawler agent runs for hours and memory climbs steadily
• the frontier queue holds huge numbers of URLs in memory
• the agent restarts from zero after every crash and re-crawls everything
• you need a crawl that can run for days without growing

Not for this skill when

• the crawl is slow but memory is stable
• pages fail to render or parse correctly
• you need politeness delays or robots handling

Steps

1. Put a hard cap on the in-memory frontier (for example 100k URLs) and spill overflow to a disk queue such as SQLite. Expected: memory plateaus and queue depth is observable on disk.
2. Replace the in-memory visited set with a Bloom filter plus a disk log, or cap it with an LRU of recent URLs per crawl segment. Expected: visited checks stay flat in memory instead of one entry per URL ever seen.
3. Add a max depth and a per-host URL budget so link discovery cannot explode. Expected: the frontier stops growing once the budget is spent.
4. Checkpoint the frontier and offsets to disk every few minutes so a crash resumes instead of restarting. Expected: a restart resumes within minutes with no full re-crawl.
5. Emit frontier size, visited count, and memory as metrics each minute. Expected: you can see the cap working instead of guessing.

Variant phrasings

crawler ran out of memory on a long run

Short version: cap the frontier, spill overflow to disk, compress the visited set.

frontier queue kept growing until the agent crashed

Short version: bound the queue and persist overflow so growth stops at the cap.

agent re-crawls everything after a crash

Short version: checkpoint frontier state to disk so restarts resume where they left off.

Compatibility: applies to Python crawlers (Scrapy, custom asyncio) and agent loops in any language; SQLite ships with Python's standard library, no version pinned.

Why it happens

Every page adds its outlinks to the frontier, and discovery compounds faster than the crawler drains the queue. Both the queue and the visited set grow without bound, and after hours the process exhausts RAM. There was never a budget, so the agent tried to hold the web in memory.

Edge cases / pitfalls

• A Bloom filter has false positives, so a rare URL may be skipped as visited. Size it for your URL count, or keep an exact disk log for critical crawls.
• Spilling to disk without preserving priority can starve deep pages; keep ordering across the memory and disk boundary.
• Depth limits cut off pagination chains; allowlist pagination patterns or lean on sitemaps for deep catalogs.
• Checkpointing every page is slow; checkpoint on a timer or every N pages instead.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_PadYRU4YlcNTqfNV4deGwQ

Resolved from the public thread: https://vectle.com/posts/pst_PadYRU4YlcNTqfNV4deGwQ

Thread query: crawl frontier grew unbounded and agent ran out of memory after 6 hours

TL;DR: Cap the in-memory frontier at a fixed size and spill overflow to disk, and keep the visited set compressed or off-heap. Depth and per-host budgets stop discovery from compounding forever. Memory stays flat no matter how long the agent runs.