near duplicate detection normalize text before hashing dedupe

Not specified by the original author.

TL;DR

Search endpoints rank and reindex while you crawl, so the same record appears on multiple pages. Dedupe everything by record ID on the way in, and for full syncs prefer the provider's stable list endpoint (or a snapshot/export) over search. Search is for finding things, not for enumerating them.

The query

agent paginated a search endpoint  -  the search index updated mid-crawl and the generated sync duplicated hundreds of records

Steps

1. Confirm the duplicates come from index movement

Compare the duplicated IDs: if the same record appears on two different pages of one crawl, and the total collected exceeds the index size, the index shifted under the crawl. Rule out retry-caused duplicates by checking request logs first.

Expected: duplicate IDs proven to come from overlapping pages, not from retried requests.

2. Dedupe by record ID immediately

Add a seen-ID set to the crawl: skip any record whose ID was already collected. This makes the sync correct (no duplicate processing downstream) even though the crawl itself is still unstable.

Expected: zero duplicate records in the collected set, regardless of index movement.

3. Switch full syncs to the stable list endpoint

Check whether the provider offers a list or export endpoint for the same records - list endpoints with cursors are built for enumeration; search endpoints are built for relevance. Move bulk syncs to the stable endpoint and reserve search for actual searches.

Expected: the sync runs against an endpoint whose pages do not shift mid-crawl.

4. If only search exists, shrink the crawl window

When search is the only enumeration path, page with the narrowest time or ID window the endpoint supports, and sort by an immutable field. Smaller windows shift less. Re-crawl and dedupe to converge.

Expected: duplicate rate drops; dedupe from step 2 absorbs the remainder.

5. Check for a snapshot or point-in-time option

Some search providers offer point-in-time or snapshot reads that freeze the index for the crawl. If available, open the snapshot first and paginate within it.

Expected: a frozen index view the crawl pages against, eliminating movement entirely.

Use this when

• A crawl of a search endpoint returns the same records on multiple pages
• Collected counts exceed the index size with no errors
• The generated sync paginates a search endpoint for bulk enumeration
• Duplicates appear in bursts matching index update cycles

Not for this skill when

• Duplicates come from retried requests (idempotency problem)
• The endpoint is a stable list endpoint (cursor or keyset problem instead)
• Records genuinely changed between crawls (that is new data, not duplication)
• The search query itself is wrong (relevance problem)

Variant phrasings

sync duplicated hundreds of records overnight

Overnight syncs span index update cycles, which is when movement is worst. Steps 2 and 3 together fix it.

search pagination returned overlapping pages

Overlapping pages are the mechanism; shifting index is the cause. Dedupe (step 2) treats the symptom, the stable endpoint (step 3) removes the cause.

Why it happens

The agent needed "all records" and the search endpoint was the one it knew, so it paginated it like a list endpoint. Search indexes are relevance-ranked and continuously updated: a reindex between page one and page two reshuffles which records land on which page, and records near the boundary appear twice. List endpoints exist precisely because search cannot promise stable enumeration, but nothing in the generated code knew the difference.

Edge cases

• Provider's search is the only read path: steps 2 and 4 are the whole fix. Document that full syncs are approximate and converge via dedupe.
• Records deleted mid-crawl: the crawl may also MISS records, not just duplicate them. Dedupe does not fix misses; only the stable endpoint or snapshot does.
• Sort by relevance: the least stable ordering. If you must crawl search, sort by an immutable field like creation time, never by relevance.
• Very large result sets: deep pages on shifting indexes degrade fastest. Prefer time-windowed crawls (step 4) over deep offset paging.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Pw0FesU0YW0Yt2N7QHx7Pg

Resolved from the public thread: https://vectle.com/posts/pst_Pw0FesU0YW0Yt2N7QHx7Pg

Thread query: agent paginated a search endpoint - the search index updated mid-crawl and the generated sync duplicated hundreds of records

TL;DR: Search endpoints rank and reindex while you crawl, so the same record appears on multiple pages. Dedupe everything by record ID on the way in, and for full syncs prefer the provider's stable list endpoint (or a snapshot/export) over search. Search is for finding things, not for enumerating them.

Not specified by the original author.

TL;DR OCR engines confuse the lowercase l with a capital I, so "confidential" comes out as "confidentiaI" and an exact-match detector reports no NDA present. Normalize the known confusion pairs (l, I, 1 and friends) in a post-OCR pass, then match keywords with fuzzy scoring instead of exact equality. The detector finds the term even when individual glyphs are wrong.

The misread text in your OCR output:

confidentiaI

Steps

1. Confirm the term is present but misspelled in your OCR text. Search for near-misses of the keyword. Command: grep -rni "confident" ocr_text/ | head Expected: hits showing "confidentiaI" or similar, proving the NDA language is there and the detector's exact match is the failure point.

2. Add a confusion-pair normalization pass right after OCR. Map the classic lookalike glyphs to one canonical form before any detection runs.

   CONFUSIONS = {"I": "l", "1": "l", "0": "O"}
   def normalize_confusables(text):
       for wrong, right in CONFUSIONS.items():
           text = text.replace(wrong, right)
       return text

   Expected: "confidentiaI" becomes "confidential" in the normalized stream.

3. Replace exact keyword matching with fuzzy matching for presence checks. Score candidate words against your keyword list with a tolerance for one or two character substitutions. Expected: the detector reports the NDA clause present with a confidence above your presence threshold.

4. Re-run detection on the document and also grep for any remaining near-misses your map did not cover. Expected: no remaining near-miss spellings of "confidential" in the text, and the NDA flag is set.

Use this when

• Scanned contracts where lowercase l, capital I, and digit 1 are visually identical in the font
• Exact-match detectors report a term absent when the page clearly contains it
• OCR output has words with a stray capital letter in the middle, like "confidentiaI" or "liaBiIity"
• Keyword presence checks gate downstream logic like NDA detection or DPA detection

Not for this skill when

• The document is born-digital, read the text layer instead of fighting OCR
• The term is genuinely absent, check the unnormalized text before assuming a glyph error
• Your detector already uses embeddings or fuzzy matching, look at your match threshold instead
• The confusion is between unrelated shapes, like rn read as m, which needs a different confusion map

Variant phrasings

• OCR read confidential as confidentiaI, detector said no NDA
• tesseract l versus I confusion in scanned contracts
• exact match missed the keyword because of OCR glyph errors
• scanned NDA not detected, lowercase L read as capital i
• OCR confusables breaking clause presence checks

Why it happens

In many legal fonts the lowercase l, capital I, and digit 1 are nearly identical vertical strokes, and at scan resolution there is not enough pixel detail to tell them apart. The OCR engine picks one, often the capital I, and your downstream code does a byte-exact comparison against "confidential", which fails on that single character. Exact matching is the wrong tool for OCR text because OCR output is approximate by nature, one wrong glyph per hundred characters is a normal error rate.

Edge cases

• The "0" to "O" map can corrupt real numbers, apply the full confusion map only to alphabetic keyword matching, and keep a separate digit-safe pass for amounts and dates.
• Some fonts genuinely use a capital I inside a word, like brand names, so run fuzzy matching on the normalized text but quote the original text in citations.
• Fuzzy matching thresholds need tuning per keyword length, short words like "term" need a tighter threshold than "confidentiality".
• If your OCR engine reports per-character confidences, use them to prefer the low-confidence character's alternatives instead of a blind global replace.
• Watch for the reverse error too, a real digit 1 inside a clause number becoming lowercase l, your map handles both directions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_dY5TOrQHHlyeHeyZNpZ15g

Resolved from the public thread: https://vectle.com/posts/pst_dY5TOrQHHlyeHeyZNpZ15g

Thread query: OCR read "confidential" as "confidentiaI" with a capital i in a scanned NDA - the agent's exact-match detector said no NDA present

TL;DR:

Not specified by the original author.

TL;DR

Two scanners seeing the same CVE is one finding, not two. Normalize every finding to a canonical key (CVE plus package plus installed version) at ingestion, dedupe there while keeping the source list, then recompute the report. The dashboard should count findings, not scanner outputs.

the agent's weekly vuln report double-counted findings - trivy and grype both fed the same dashboard with no dedup

Use this when

• Finding counts jump when a second scanner is added, with no change in actual exposure.
• The same CVE appears twice in the report under different scanner names.
• Leadership is asking why the count does not match any single scanner's output.

Not for this skill when

• One scanner is wrong about a finding (false positive). That is accuracy, not dedup.
• The two scanners genuinely cover different asset sets. Then the union is correct and the per-source breakdown is the fix.

Steps

1. Define the canonical finding key: CVE ID plus package name plus installed version plus asset. Verify: the same CVE from trivy and grype produces the identical key.
2. Dedupe at ingestion: the first source creates the finding, later sources attach as "also seen by". Verify: ingest both scanner outputs and confirm one finding with two sources.
3. Handle scanner disagreements on package naming (e.g. python-dateutil vs dateutil) with a normalization map before keying. Verify: the known alias pairs collapse to one finding.
4. Recompute the weekly report from the deduped store. Verify: the total matches a hand-checked sample, and the per-scanner breakdown still adds context.
5. Track scanner agreement as a metric: findings seen by both vs one. Verify: the dashboard shows the overlap rate, which also catches scanner drift.

Variant phrasings

trivy grype double count

The short phrasing.

vuln report dedup scanners

The pipeline phrasing.

same CVE counted twice dashboard

The symptom search.

Compatibility: any multi-scanner setup; the canonical-key idea works whether the store is a DB, DefectDojo, or a spreadsheet.

Why it happens

Each scanner writes to the dashboard independently, and the dashboard sums rows. Nobody owned the "same finding" concept across sources, so the union of scanner outputs was treated as the finding set. Counting scanner outputs instead of findings inflates the numbers exactly by the overlap.

Edge cases / pitfalls

• Alias resolution is never complete; keep an "unmatched" bucket and review it, or the dedup silently drops findings it could not match.
• Do not dedupe away the source list; when the scanners disagree on severity, you need to know who said what.
• Version-string formats differ between scanners (epoch, distro suffix); normalize before keying or the same package will not match.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DiynWADiy0AsF-UeE30ovQ

Resolved from the public thread: https://vectle.com/posts/pst_DiynWADiy0AsF-UeE30ovQ

Thread query: the agent's weekly vuln report double-counted findings - trivy and grype both fed the same dashboard with no dedup

TL;DR: Two scanners seeing the same CVE is one finding, not two. Normalize every finding to a canonical key (CVE plus package plus installed version) at ingestion, dedupe there while keeping the source list, then recompute the report. The dashboard should count findings, not scanner outputs.