intel agent failed on syndicated news dedup across wire services

Intel agent failed on syndicated news dedup across wire services

TL;DR

Wire-service dedup fails when the agent compares headlines, because wires rewrite them. Dedupe on the story's fingerprint instead: normalized lead paragraph plus publication date window. Cluster first, keep the cleanest copy, and record the wire variants as duplicates. The briefing carries one story per event.

The error

(duplicates in briefing)
intel agent failed on syndicated news dedup across wire services; same story 5 times from 5 wires

When this helps

• briefings repeat wire stories
• multi-wire intake produces duplicates
• building story deduping
• validating briefing uniqueness

When it doesn't

• wires carry genuinely different angles; those are separate stories
• updates add material facts; version them, do not merge
• you need per-wire attribution; dedupe conflicts with that goal

Works with

python 3.8+ with hashlib and re. No external dependencies.

Steps

1. Fingerprint each story on normalized content

import hashlib, re
def story_key(text, date):
    norm = re.sub(r"\s+", " ", text.strip().lower())[:300]
    return hashlib.md5((date + norm).encode()).hexdigest()
print(story_key("Markets rally on earnings.", "2026-10-09"))

Expected: A content fingerprint. Headline edits across wires do not change it.

2. Cluster stories by fingerprint

import hashlib, re
def story_key(text, date):
    norm = re.sub(r"\s+", " ", text.strip().lower())[:300]
    return hashlib.md5((date + norm).encode()).hexdigest()
stories = [("Markets rally on earnings.", "2026-10-09", "wire-a"), ("Markets rally on earnings!", "2026-10-09", "wire-b")]
clusters = {}
for text, date, wire in stories:
    clusters.setdefault(story_key(text, date), []).append(wire)
print("clusters:", len(clusters), "for", len(stories), "stories")

Expected: One cluster per real story. Wire variants collapse correctly.

3. Keep the fullest copy per cluster

import json
best = {"key": "abc", "wire": "wire-a", "dupes": ["wire-b", "wire-c"]}
open("stories.json", "w").write(json.dumps(best, indent=2))
print("fullest copy kept; variants recorded as dupes")

Expected: A canonical story record. The briefing reads the best copy once.

4. Validate uniqueness before publishing

import json
items = [{"key": "a"}, {"key": "a"}, {"key": "b"}]
keys = [i["key"] for i in items]
print("dupes remaining:", len(keys) != len(set(keys)))

Expected: A pre-publish check. Any True means the fingerprinter missed a variant.

Other ways people phrase this

wire service dedup failed

Fingerprint the normalized lead, not the headline.

syndicated news duplicates agent

Cluster by content, keep the fullest copy.

same story multiple wires briefing

One story per event. Variants are dupes, not coverage.

Why it happens

Wires syndicate the same story with different headlines, templates, and trims. Headline or URL deduping misses every variant. The story's identity is its content plus date; fingerprinting normalized text catches what string matching cannot.

Edge cases

• Developing stories update through the day; fingerprint per version with timestamps.
• Opinion pieces on the same event are different stories; keep them separate.
• Very short alerts fingerprint poorly; require minimum length.
• Date windows matter; the same text a week later may be a new development.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9Nwh1dvH1d4JN0IAa472XQ