agent failed to detect the duplicate: cached the stale company list

TL;DR

The agent cached the company list hours ago and matches new records against stale data, so recently created companies never match. Set a short TTL on the cache, invalidate it whenever the agent writes companies, and refresh before each matching run. The matching logic is fine; its input is old.

Error

(No error; the symptom is duplicates the agent should have caught, all involving recently created companies.)

Steps

1. Check the cache timestamp versus the missed duplicates' creation times. Expected: the cache predates them.
2. Set a TTL on the company cache (minutes, not hours, for active orgs). Expected: staleness bounded.
3. Invalidate the cache on every company write the agent performs. Expected: the agent always sees its own writes.
4. Refresh the cache at the start of each matching run regardless of TTL. Expected: runs start from current data.
5. Log cache age with match results for audit. Expected: future misses are diagnosable.

When to use

• Duplicates missed involving recent records.
• An agent caches reference data for performance.
• After bulk imports, matching degrades.

When not to use

• Matching logic errors (normalize and block first).
• Real-time requirements (then do not cache at all).

Tool compatibility

• Any agent-side cache; HubSpot and Salesforce list APIs.
• TTL and invalidation logic.

Variant phrasings

stale cache missed duplicates

Shorten TTL; invalidate on write.

matching against old data

Refresh before each run.

Why it happens

Caching trades freshness for speed. Dedupe is exactly the workload where freshness matters most, so the tradeoff bites hardest here.

Edge cases

• Very active orgs need TTLs in minutes; quiet orgs can use longer.
• Writes from other systems do not invalidate your cache; TTL is the backstop.
• Cache stampedes on refresh can spike API usage; refresh once per run, not per record.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstUpK7fDFVOB0dNTnV6nXeA