intel agent loop detected re-downloading the same sec filing, dedup failed

Intel agent loop detected re-downloading the same SEC filing

TL;DR

Re-downloading the same filing in a loop means the agent never records that the download finished. The fix is a download ledger keyed by accession: check the ledger before fetching, record immediately after. Deduping the queue at startup removes the repeats that feed the loop.

The error

(agent loop detected)
intel agent loop detected re-downloading the same sec filing; dedup failed, bandwidth wasted

When this helps

• an agent re-downloads the same filings
• loop detectors fire on download patterns
• building idempotent download queues
• debugging agent download loops

When it doesn't

• re-download is intentional, like refreshing a restated filing; version the entries
• the loop is in reasoning, not downloads
• the queue source duplicates; fix the source too

Works with

python 3.8+ with json and os. No external dependencies.

Steps

1. Ledger every download by accession

import json
ledger = {}
def mark_downloaded(acc):
    ledger[acc] = "downloaded"
    open("dl_ledger.json", "w").write(json.dumps(ledger))
mark_downloaded("[acc1]")
print(open("dl_ledger.json").read())

Expected: A download ledger. The agent checks it before every fetch.

2. Skip already-downloaded accessions

import json, os
ledger = json.load(open("dl_ledger.json"))
for acc in ["[acc1]", "[acc2]"]:
    if acc in ledger:
        print(acc, "skipped: already downloaded")
    else:
        print(acc, "fetching")

Expected: No repeat downloads. The loop cannot form when fetched items are skipped.

3. Dedupe the download queue at startup

import json
queue = ["[acc1]", "[acc2]", "[acc1]"]
ledger = json.load(open("dl_ledger.json"))
seen = set(ledger)
unique = []
for a in queue:
    if a not in seen:
        seen.add(a)
        unique.append(a)
print("unique to fetch:", unique)

Expected: A deduped queue. Input duplicates never reach the fetcher.

4. Alert when the same accession is requested twice in one run

import json
requested = []
def request(acc):
    if acc in requested:
        print("LOOP WARNING:", acc, "requested twice")
    requested.append(acc)
request("[acc1]")
request("[acc1]")

Expected: A loop warning. Double requests in one run signal a logic bug upstream.

Other ways people phrase this

agent re-downloading same filing loop

Ledger by accession. Check before fetch, record after.

dedup failed download queue agent

Dedupe at startup into a dict-keyed ledger.

sec filing download loop detected

The ledger makes repeats impossible; the warning catches logic bugs.

Why it happens

Agents loop on downloads when fetch completion is not recorded. Each iteration sees the same pending list because nothing marks items done. A persistent ledger keyed by accession turns downloads into a checklist, and dict keys cannot duplicate.

Edge cases

• Ledger writes must be atomic; torn writes resurrect the loop.
• Restated filings need re-download; mark the old entry superseded, not done.
• A download that 404s should be marked failed, not left pending.
• Share the ledger across agents with file locking to avoid races.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aRBZnCkUMRina8bQ0c6CRQ