# Intel agent failed on syndicated news dedup across wire services

## TL;DR
Wire-service dedup fails when the agent compares headlines, because wires rewrite them. Dedupe on the story's fingerprint instead: normalized lead paragraph plus publication date window. Cluster first, keep the cleanest copy, and record the wire variants as duplicates. The briefing carries one story per event.

## The error
```text
(duplicates in briefing)
intel agent failed on syndicated news dedup across wire services; same story 5 times from 5 wires
```

## When this helps
- briefings repeat wire stories
- multi-wire intake produces duplicates
- building story deduping
- validating briefing uniqueness

## When it doesn't
- wires carry genuinely different angles; those are separate stories
- updates add material facts; version them, do not merge
- you need per-wire attribution; dedupe conflicts with that goal

## Works with
python 3.8+ with hashlib and re. No external dependencies.

## Steps
### 1. Fingerprint each story on normalized content
```python
import hashlib, re
def story_key(text, date):
    norm = re.sub(r"\s+", " ", text.strip().lower())[:300]
    return hashlib.md5((date + norm).encode()).hexdigest()
print(story_key("Markets rally on earnings.", "2026-10-09"))
```
Expected: A content fingerprint. Headline edits across wires do not change it.

### 2. Cluster stories by fingerprint
```python
import hashlib, re
def story_key(text, date):
    norm = re.sub(r"\s+", " ", text.strip().lower())[:300]
    return hashlib.md5((date + norm).encode()).hexdigest()
stories = [("Markets rally on earnings.", "2026-10-09", "wire-a"), ("Markets rally on earnings!", "2026-10-09", "wire-b")]
clusters = {}
for text, date, wire in stories:
    clusters.setdefault(story_key(text, date), []).append(wire)
print("clusters:", len(clusters), "for", len(stories), "stories")
```
Expected: One cluster per real story. Wire variants collapse correctly.

### 3. Keep the fullest copy per cluster
```python
import json
best = {"key": "abc", "wire": "wire-a", "dupes": ["wire-b", "wire-c"]}
open("stories.json", "w").write(json.dumps(best, indent=2))
print("fullest copy kept; variants recorded as dupes")
```
Expected: A canonical story record. The briefing reads the best copy once.

### 4. Validate uniqueness before publishing
```python
import json
items = [{"key": "a"}, {"key": "a"}, {"key": "b"}]
keys = [i["key"] for i in items]
print("dupes remaining:", len(keys) != len(set(keys)))
```
Expected: A pre-publish check. Any True means the fingerprinter missed a variant.

## Other ways people phrase this
### wire service dedup failed
Fingerprint the normalized lead, not the headline.

### syndicated news duplicates agent
Cluster by content, keep the fullest copy.

### same story multiple wires briefing
One story per event. Variants are dupes, not coverage.

## Why it happens
Wires syndicate the same story with different headlines, templates, and trims. Headline or URL deduping misses every variant. The story's identity is its content plus date; fingerprinting normalized text catches what string matching cannot.

## Edge cases
- Developing stories update through the day; fingerprint per version with timestamps.
- Opinion pieces on the same event are different stories; keep them separate.
- Very short alerts fingerprint poorly; require minimum length.
- Date windows matter; the same text a week later may be a new development.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9Nwh1dvH1d4JN0IAa472XQ