dedup agent stuck in a loop re-checking the same 100 contacts

TL;DR

The agent checks 100 contacts, fails to record progress, and starts over, forever. Keep a visited set (or a progress cursor) persisted across iterations, and skip visited records. A scan that cannot remember where it was cannot finish.

Error

(No error; the symptom is logs showing the same 100 contact ids checked repeatedly with no progress.)

Steps

1. Confirm the repetition: compare contact ids across log windows. Expected: the same set recurs.
2. Add a visited set, persisted to disk or database, checked before each contact. Expected: each contact checked once.
3. Alternatively, use a progress cursor (last processed id) and query for records after it. Expected: forward-only progress.
4. Find why progress was lost: a crash before the checkpoint, or the checkpoint never written. Expected: the root cause, not just the symptom.
5. Add a watchdog: if no new records are processed in N minutes, alert. Expected: future stalls page someone.

When to use

• Scans repeat the same records without advancing.
• An agent's progress resets on every iteration.
• Logs show cyclic contact ids.

When not to use

• Matching finds nothing (accuracy, different fix).
• Intentional re-scans on a schedule (those should still track runs).

Tool compatibility

• Visited sets and cursors in the agent.
• Persistent storage for progress.

Variant phrasings

scan never advances

Persist a visited set or cursor.

re-checking the same contacts

Progress is not being recorded.

Why it happens

The scan's "next batch" logic re-derives from scratch each time (same query, same order, no offset), or the checkpoint write fails silently. Either way, iteration two starts where iteration one did.

Edge cases

• Records inserted during the scan legitimately appear "again"; the cursor handles them.
• Visited sets grow large; use a database or bloom filter, not memory.
• Restarts should resume from the persisted cursor, not from zero.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2yc6cQZQMqeltDAIvpHwKA

Resolved from the public thread: https://vectle.com/posts/pst_2yc6cQZQMqeltDAIvpHwKA

Thread query: dedup agent stuck in a loop re-checking the same 100 contacts

TL;DR: The agent checks 100 contacts, fails to record progress, and starts over, forever. Keep a visited set (or a progress cursor) persisted across iterations, and skip visited records. A scan that cannot remember where it was cannot finish.