intel agent crashed tracking 200 competitor skus, memory error

Intel agent crashed tracking 200 competitor SKUs, memory error

TL;DR

Tracking 200 SKUs crashes the agent when it holds every product page, price history, and comparison in memory at once. The fix is streaming: process one SKU at a time, persist its record, and release the memory before the next. The SKU list becomes a queue, and the run survives any single SKU's failure.

The error

(agent crashed)
intel agent crashed tracking 200 competitor skus, memory error; run died at sku 140

When this helps

• an agent crashes tracking large SKU lists
• competitor catalog sweeps die partway
• building streaming SKU pipelines
• adding resumability to catalog runs

When it doesn't

• the catalog is small; streaming adds file overhead
• you need cross-SKU analysis live; stream then analyze the records
• the crash is a bug in one SKU's parser; fix the parser

Works with

python 3.8+ with json. No external dependencies.

Steps

1. Stream SKUs one at a time from the queue

import json
skus = ["sku-%d" % i for i in range(200)]
for sku in skus:
    record = {"sku": sku, "price": "pending"}
    open("sku_" + sku + ".json", "w").write(json.dumps(record))
print("200 small records, never 200 pages in memory")

Expected: Per-SKU record files. Memory holds one SKU at a time, not the catalog.

2. Persist each SKU before moving on

import json
rec = json.load(open("sku_sku-1.json"))
rec["price"] = "$99"
json.dump(rec, open("sku_sku-1.json", "w"))
print("persisted; a crash loses at most one sku")

Expected: Immediate persistence. The crash at SKU 140 becomes a resume at 140, not a restart.

3. Checkpoint the queue position

import json
open("sku_progress.json", "w").write(json.dumps({"done": 140, "total": 200}))
print("resume from sku 141")

Expected: A progress file. Reruns skip the 140 completed SKUs.

4. Cap per-SKU work with a timeout

print("wrap each sku fetch in a 60s timeout")
print("on timeout: mark the sku failed, continue")

Expected: A per-SKU timeout. One slow product page cannot stall the catalog.

Other ways people phrase this

agent crashed tracking skus memory

One SKU in memory at a time. Persist per SKU. Checkpoint the queue.

competitor sku sweep died

Resume from the checkpoint. The crash loses one SKU, not the run.

200 skus out of memory agent

Streaming plus per-SKU timeouts. No step holds the catalog.

Why it happens

Agents that accumulate scraped pages and histories for hundreds of SKUs exhaust memory linearly with catalog size. Streaming bounds memory to one SKU, per-SKU persistence bounds loss to one SKU, and checkpoints make reruns cheap. The catalog size stops mattering.

Edge cases

• Per-SKU files multiply; archive them into one catalog file at the end.
• Price histories grow unbounded; cap history length per SKU.
• A SKU that fails repeatedly should be marked failed, not retried forever.
• Respect the site's rate limits across the whole sweep; 200 fast requests invite bans.
• SKU pages that change layout mid-sweep break the parser for later SKUs; re-validate the parser every 25 SKUs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sDKvJp3WZnFOBkCuwA5r-g