# Intel agent crashed tracking 200 competitor SKUs, memory error

## TL;DR
Tracking 200 SKUs crashes the agent when it holds every product page, price history, and comparison in memory at once. The fix is streaming: process one SKU at a time, persist its record, and release the memory before the next. The SKU list becomes a queue, and the run survives any single SKU's failure.

## The error
```text
(agent crashed)
intel agent crashed tracking 200 competitor skus, memory error; run died at sku 140
```

## When this helps
- an agent crashes tracking large SKU lists
- competitor catalog sweeps die partway
- building streaming SKU pipelines
- adding resumability to catalog runs

## When it doesn't
- the catalog is small; streaming adds file overhead
- you need cross-SKU analysis live; stream then analyze the records
- the crash is a bug in one SKU's parser; fix the parser

## Works with
python 3.8+ with json. No external dependencies.

## Steps
### 1. Stream SKUs one at a time from the queue
```python
import json
skus = ["sku-%d" % i for i in range(200)]
for sku in skus:
    record = {"sku": sku, "price": "pending"}
    open("sku_" + sku + ".json", "w").write(json.dumps(record))
print("200 small records, never 200 pages in memory")
```
Expected: Per-SKU record files. Memory holds one SKU at a time, not the catalog.

### 2. Persist each SKU before moving on
```python
import json
rec = json.load(open("sku_sku-1.json"))
rec["price"] = "$99"
json.dump(rec, open("sku_sku-1.json", "w"))
print("persisted; a crash loses at most one sku")
```
Expected: Immediate persistence. The crash at SKU 140 becomes a resume at 140, not a restart.

### 3. Checkpoint the queue position
```python
import json
open("sku_progress.json", "w").write(json.dumps({"done": 140, "total": 200}))
print("resume from sku 141")
```
Expected: A progress file. Reruns skip the 140 completed SKUs.

### 4. Cap per-SKU work with a timeout
```python
print("wrap each sku fetch in a 60s timeout")
print("on timeout: mark the sku failed, continue")
```
Expected: A per-SKU timeout. One slow product page cannot stall the catalog.

## Other ways people phrase this
### agent crashed tracking skus memory
One SKU in memory at a time. Persist per SKU. Checkpoint the queue.

### competitor sku sweep died
Resume from the checkpoint. The crash loses one SKU, not the run.

### 200 skus out of memory agent
Streaming plus per-SKU timeouts. No step holds the catalog.

## Why it happens
Agents that accumulate scraped pages and histories for hundreds of SKUs exhaust memory linearly with catalog size. Streaming bounds memory to one SKU, per-SKU persistence bounds loss to one SKU, and checkpoints make reruns cheap. The catalog size stops mattering.

## Edge cases
- Per-SKU files multiply; archive them into one catalog file at the end.
- Price histories grow unbounded; cap history length per SKU.
- A SKU that fails repeatedly should be marked failed, not retried forever.
- Respect the site's rate limits across the whole sweep; 200 fast requests invite bans.
- SKU pages that change layout mid-sweep break the parser for later SKUs; re-validate the parser every 25 SKUs.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sDKvJp3WZnFOBkCuwA5r-g