intel agent stuck in loop re-summarizing the same filing, dedup failed

Intel agent stuck in a loop re-summarizing the same filing

TL;DR

A re-summarizing loop means the agent has no memory of what it already did: it fetches, summarizes, and repeats because nothing records the completion. The fix is a work ledger: every filing gets a state, and the agent only processes filings in the pending state. Deduping the queue at the start prevents the loop from ever forming.

The error

(agent loop)
intel agent stuck re-summarizing the same filing; dedup failed, run never progresses

When this helps

• an agent re-processes the same filing repeatedly
• runs never progress past the first few filings
• building idempotent agent work queues
• debugging agent loops

When it doesn't

• the loop is in reasoning, not work items; that needs a different guard
• re-summarization is intentional, like a nightly refresh; then version the outputs
• the queue source itself duplicates; fix the source too

Works with

python 3.8+ with json. No external dependencies.

Steps

1. Keep a ledger of filing states

import json
ledger = {"[acc1]": "summarized", "[acc2]": "pending", "[acc3]": "pending"}
open("ledger.json", "w").write(json.dumps(ledger, indent=2))
print("ledger:", open("ledger.json").read())

Expected: A state file. The agent's next action is always the oldest pending filing, never a repeat.

2. Dedupe the work queue before starting

import json
queue = ["[acc1]", "[acc2]", "[acc1]", "[acc3]"]
ledger = json.load(open("ledger.json"))
pending = []
for a in queue:
    if a not in ledger and a not in pending:
        pending.append(a)
        ledger[a] = "pending"
json.dump(ledger, open("ledger.json", "w"), indent=2)
print("deduped pending:", pending)

Expected: A deduped queue merged into the ledger. Duplicates in the input never become duplicate work.

3. Mark each filing done immediately after summarizing

import json
ledger = json.load(open("ledger.json"))
ledger["[acc2]"] = "summarized"
json.dump(ledger, open("ledger.json", "w"), indent=2)
print("marked done right after the summary, before the next fetch")

Expected: Immediate state updates. Marking done before the next step means even a crash cannot repeat the work.

4. Add a loop guard that halts on repeats

import json
ledger = json.load(open("ledger.json"))
counts = {}
for acc, state in ledger.items():
    counts[acc] = counts.get(acc, 0) + 1
print("ledger is a dict; duplicate keys are impossible by construction")

Expected: A structural guarantee. A dict ledger cannot hold the same filing twice, which makes the loop class impossible.

Other ways people phrase this

agent loop re-summarizing filing

A work ledger with states. Pending only, never repeat.

dedup failed agent loop

Dedupe the queue into the ledger at startup. Dict keys cannot duplicate.

intel agent stuck same filing

Mark done immediately after each unit of work. Crash-safe by construction.

Why it happens

Agents loop on work when completion is not recorded anywhere. Each iteration sees the same pending work because nothing distinguishes done from todo. A persistent ledger with explicit states turns the queue into a checklist, and dict-keyed storage makes duplicates structurally impossible.

Edge cases

• Ledger writes must be atomic; a torn write resurrects the loop.
• Long runs should compact the ledger; done entries older than a week can archive out.
• If two agents share a ledger, use file locking or a real queue.
• A filing that fails summarizing should be marked failed, not left pending forever.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstAol9HWmUHw97uvJrR5yfw