the agent hallucinated blocklist entries that were never published; downstream filter sync failed
Stops an agent hallucinating blocklist entries that were never published, by requiring entries to cite a verified indicator source before sync. Use when downstream filters receive block entries with no corresponding approved indicator. Not for legitimate entries whose publication record is merely misfiled.
TL;DR
A blocklist is a weapon: a fabricated entry blocks real mail or real sites, and a model that invents plausible domains will eventually invent a customer domain. Every entry must trace to evidence: a message, a case or an analyst approval, with the identifier recorded. Enforce that in code: the sync step rejects any entry without a source reference, and entries are generated from the indicator table, never from free text in a report.
the agent hallucinated blocklist entries that were never published; downstream filter sync failedUse this when
- Filter sync applied entries nobody can find in any case or report.
- An entry blocks a legitimate domain and no analyst recalls adding it.
Not for this skill when
- The entry is real but was published under a different system name. Reconcile records before redesigning.
- Entries are correct but the sync duplicates them. That is a sync mechanics issue.
Steps
- Audit the current blocklist against source records and list entries with no traceable source. Verify: the unsourced set is enumerated.
- Remove unsourced entries after a quick analyst review for any that correspond to live campaigns. Verify: the list now contains only evidenced entries.
- Change the generation path: blocklist entries are built only from the indicator table, each row carrying its case or message reference. Verify: free-text report content can no longer become an entry.
- Add a sync gate that rejects entries missing a source reference and logs the rejection. Verify: a planted unsourced entry is refused in testing.
- Monitor filter impact for a week after cleanup. Verify: no legitimate mail is blocked by a phantom entry, and real campaign entries still apply.
Variant phrasings
agent hallucinated blocklist entries never published sync
The phantom entry variant.
blocklist entries require source evidence indicator table
The evidence gate variant.
Compatibility: Tenant block lists and threat indicator pipelines fed by agent triage.
Why it happens
Generation is fluent at producing domain-shaped strings, which is exactly why generation must never be the source of enforcement data. The blocklist is an act with consequences for third parties, and acts need provenance.
Edge cases / pitfalls
- Apply the same provenance rule to allowlist entries, where invention is equally damaging in the other direction.
- Keep the rejection log; what the model tried to add is useful signal about its failure modes.
- Human approval for entries affecting high-traffic domains is worth the friction. Automate the small, review the big.
Provenance
Resolved from the public thread: https://vectle.com/posts/pstY7u3I_42j8PSv5e3O17Pw