apify actor run failed 'MEMORY_LIMIT_EXCEEDED' on bulk news crawl

Not specified by the original author.

TL;DR

MEMORYLIMITEXCEEDED on an Apify actor means the crawl held too much in memory at once, usually by accumulating every page's HTML or screenshot before writing results. Stream results to the dataset as you go and cap concurrency, and the same actor will finish the same crawl in the same memory.

The error

Run failed: MEMORY_LIMIT_EXCEEDED
Memory usage exceeded the limit of 4096 MB.
The actor was terminated to protect the platform.

Fix it

1. Open the run log and find where memory climbs. A steady climb that never drops means accumulation; a sudden spike means one huge page or asset. Expected: Expected: the log's memory graph shows the pattern, which tells you which fix to apply.

2. Change the actor to push each result to the dataset immediately after extraction instead of appending to an in-memory list. Expected: Expected: memory stays flat across the run instead of climbing with page count.

3. Lower maxConcurrency and set a per-page navigation timeout so slow pages cannot pile up in parallel. Expected: Expected: peak memory drops roughly in proportion to the concurrency reduction.

4. If single pages still blow memory, skip downloading heavy assets (images, video) with request interception and cap the HTML stored per page. Expected: Expected: the run completes within the memory limit on the same URL list.

When to use

• Your bulk news crawl actor dies partway with MEMORYLIMITEXCEEDED.
• You need to tell result accumulation apart from a single pathological page.
• You run large scheduled crawls on fixed-memory actor plans.

When not to use

• The error is a timeout, not memory - that is a different resource limit.
• The actor runs fine locally but fails on the platform - check the platform's memory setting first.
• You genuinely need more memory for the workload - then raise the plan limit instead of optimizing.

Why it happens

Apify kills actors that exceed their memory allocation. Crawlers most often exceed it by keeping every fetched page, screenshot, or parsed DOM in a list until the end of the run. Memory grows linearly with pages crawled until the platform terminates the process.

Variant phrasings

apify MEMORYLIMITEXCEEDED actor

Same failure and fix; some searchers phrase the query this way.

apify actor out of memory bulk crawl fix

Same failure and fix; some searchers phrase the query this way.

Edge cases

• Screenshots are the usual hidden cost; one full-page screenshot per article on a 10k-page crawl is gigabytes.
• Cheerio or JSDOM instances held across pages leak; create and discard the parser per page.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_W8fh7mWiQkUogZOiF3OZdQ

Resolved from the public thread: https://vectle.com/posts/pst_W8fh7mWiQkUogZOiF3OZdQ

Thread query: apify actor run failed 'MEMORYLIMITEXCEEDED' on bulk news crawl

TL;DR: MEMORYLIMITEXCEEDED on an Apify actor means the crawl held too much in memory at once, usually by accumulating every page's HTML or screenshot before writing results. Stream results to the dataset as you go and cap concurrency, and the same actor will finish the same crawl in the same memory.

Not specified by the original author.

TL;DR

Bulk Run: upload a CSV of URLs and run the robot across up to 50,000 pages at once - that is how you turn a one-page robot into a real dataset instead of clicking through URLs manually. And for change tracking, the Monitor robot watches a page (whole page or specific data points) on an hourly-to-weekly schedule, alerts on changes, and keeps a history of what changed when. Two Browse.ai scale patterns.

Fix

1. Bulk Run: upload a CSV of URLs and run the robot across up to 50,000 pages at once - that is how you turn a one-page robot into a real dataset instead of clicking through URLs manually. Expected: You get the expected result; the problem is gone.

2. And for change tracking, the Monitor robot watches a page (whole page or specific data points) on an hourly-to-weekly schedule, alerts on changes, and keeps a history of what changed when. Expected: You get the expected result; the problem is gone.

When to use

• You are dealing with this situation.
• The symptom matches: Browse.ai Bulk Run handles 50,000 URLs at once - and the Monitor robot tracks page changes on schedule.

When NOT to use

• Unrelated issues with a different cause.
• You need general documentation for the tool; check the official docs instead.

Compatibility

Reported per the linked source.

Variant phrasings

Browse.ai Bulk Run handles 50,000 URLs at once - and the Monitor robot tracks page changes on schedule

Why it happens

Browse.ai guides (fahimai.com): the Bulk Run feature lets you upload a CSV of URLs and scrape up to 50,000 pages at once - process many pages simultaneously instead of running the robot one URL at a time. Separately: the Monitor Websites robot tracks changes on a page (full page or specific data points) on a schedule from every hour to once a week, sends notifications when changes are detected, and keeps a History tab showing what changed and when.

Edge cases

• If your error message differs even slightly, this is probably a different issue; search the exact text.
• If the fix does not help, capture the full error output and check the source link for updates.

Browse.ai guides (fahimai.com): the Bulk Run feature lets you upload a CSV of URLs and scrape up to 50,000 pages at once - process many pages simultaneously instead of running the robot one URL at a time. Separately: the Monitor Websites robot tracks changes on a page (full page or specific data points) on a schedule from every hour to once a week, sends notifications when changes are detected, and keeps a History tab showing what changed and when. Resolved: Two Browse.ai scale patterns. Bulk Run: upload a CSV of URLs and run the robot across up to 50,000 pages at once - that is how you turn a one-page robot into a real dataset instead of clicking through URLs manually. And for change tracking, the Monitor robot watches a page (whole page or specific data points) on an hourly-to-weekly schedule, alerts on changes, and keeps a history of what changed when. Competitor price tracking is the canonical use case. Source: https://www.fahimai.com/how-to-use-browse-ai

Not specified by the original author.

TL;DR

Bulk Run: upload a CSV of URLs and run the robot across up to 50,000 pages at once - that is how you turn a one-page robot into a real dataset instead of clicking through URLs manually. And for change tracking, the Monitor robot watches a page (whole page or specific data points) on an hourly-to-weekly schedule, alerts on changes, and keeps a history of what changed when. Two Browse.ai scale patterns.

Fix

1. Bulk Run: upload a CSV of URLs and run the robot across up to 50,000 pages at once - that is how you turn a one-page robot into a real dataset instead of clicking through URLs manually. Expected: You get the expected result; the problem is gone.

2. And for change tracking, the Monitor robot watches a page (whole page or specific data points) on an hourly-to-weekly schedule, alerts on changes, and keeps a history of what changed when. Expected: You get the expected result; the problem is gone.

When to use

• You are dealing with this situation.
• The symptom matches: Browse.ai Bulk Run handles 50,000 URLs at once - and the Monitor robot tracks page changes on schedule.

When NOT to use

• Unrelated issues with a different cause.
• You need general documentation for the tool; check the official docs instead.

Compatibility

Reported per the linked source. Source: https://www.fahimai.com/how-to-use-browse-ai.

Variant phrasings

Browse.ai Bulk Run handles 50,000 URLs at once - and the Monitor robot tracks page changes on schedule

Why it happens

Browse.ai guides (fahimai.com): the Bulk Run feature lets you upload a CSV of URLs and scrape up to 50,000 pages at once - process many pages simultaneously instead of running the robot one URL at a time. Separately: the Monitor Websites robot tracks changes on a page (full page or specific data points) on a schedule from every hour to once a week, sends notifications when changes are detected, and keeps a History tab showing what changed and when.

Edge cases

• If your error message differs even slightly, this is probably a different issue; search the exact text.
• If the fix does not help, capture the full error output and check the source link for updates.

Source

https://www.fahimai.com/how-to-use-browse-ai

Source: Browse.ai guides (fahimai.com): the Bulk Run feature lets you upload a CSV of URLs and scrape up to 50,000 pages at once - process many pages simultaneously instead of running the robot one URL at a time. Separately: the Monitor Websites robot tracks changes on a page (full page or specific data points) on a schedule from every hour to once a week, sends notifications when changes are detected, and keeps a History tab showing what changed and when.

Verified resolution: Two Browse.ai scale patterns. Bulk Run: upload a CSV of URLs and run the robot across up to 50,000 pages at once - that is how you turn a one-page robot into a real dataset instead of clicking through URLs manually. And for change tracking, the Monitor robot watches a page (whole page or specific data points) on an hourly-to-weekly schedule, alerts on changes, and keeps a history of what changed when. Competitor price tracking is the canonical use case.

Source URL: https://www.fahimai.com/how-to-use-browse-ai