agent's ioc matching failed because defanged hxxps urls were never normalized before threat feed lookup

TL;DR

Normalise every indicator to its real form before any lookup: refang hxxps and bracketed dots and colons back to a standard URL, lowercase the host, strip trailing punctuation and tracking noise, and only then query the threat feed, logging both the original and the normalised form. Analysts and users defang URLs precisely so they are safe to share, and an agent that looks up the safe form is searching for a string that exists in no feed on earth.

agent's ioc matching failed because defanged hxxps urls were never normalized before threat feed lookup

Use this when

• The agent reports no match for URLs that a manual feed search finds immediately.
• Source material contains defanged forms such as hxxps, bracketed dots, or spaced punctuation.
• The same agent matches plain URLs from other sources correctly.

Not for this skill when

• Lookups return errors or timeouts from the feed. That is a feed connectivity case, not normalisation.
• The URL is genuinely new and in no feed. Normalisation will not create a match that does not exist.

Steps

1. Take a missed URL from a real report and write down its exact source form and the feed form that matches manually. Work from a copied message or a test send first so production mail stays untouched while you compare the before and after state. Record the timestamp, the sending service, and the exact header or log line that changed, because deliverability and phishing faults often look identical until that evidence is lined up side by side in one place. Verify: the source and matched forms are captured side by side
2. Build the normalisation step to handle the defang styles your sources actually use: hxxps, bracket dots, bracket colons, and whitespace tricks, in a fixed order. Do not change more than one setting at a time here. If you fix the record, the connector, and the policy in the same window you will not know which change cleared the fault, and the next regression will take just as long to isolate. Save the working configuration once this step checks out before moving on. Verify: the normaliser converts a test set of defanged URLs to their correct real forms
3. Canonicalise after refanging: parse the URL, normalise host case and default ports, and strip sentence punctuation stuck to the end. Use the receiving side view for this check rather than the sender console alone. Sender dashboards often report a local success while the recipient gateway applied a different verdict, rewrote a header, or dropped the message at a later hop. The recipient header, portal log, or message trace entry is the evidence that counts. Verify: canonical outputs for the test set match what the feed expects
4. Insert normalisation before every feed lookup in the agent and log original, normalised, and verdict together. Keep any exception you create narrow and time boxed. Scope it to the one sender, domain, or message type you are fixing, note the owner and the review date in the change record, and remove it once the root fix is in place so a temporary workaround does not quietly become a permanent hole in the filter. Verify: agent logs for a test report show all three values
5. Replay a batch of previously missed reports through the fixed agent and count how many now match feed entries. Retest with a fresh message through the exact same path as the failing one, not a shortcut path. A new test from a different client, gateway, or mailbox can pass while the original route is still broken, which is the most common reason these faults get closed early and then reopen within a day. Verify: the replay shows the previously missed matches now found

Why it happens

Defanging is a human safety convention, and feeds store the live dangerous form. The gap between the two is small, deterministic, and entirely mechanical, which makes it an ideal normalisation step and a frustrating bug: every individual piece works, the feed is up, the URL is known bad, and the match still fails because the strings differ. Agents that ingest analyst notes, forum posts, and user reports will meet defanged indicators constantly, so the conversion belongs in the ingestion path, not in analyst memory.

Variants and edge cases

agent ioc matching failed defanged hxxps urls never normalized threat feed

The exact normalisation gap wording.

refang hxxps bracket dot url before lookup agent fix

The refang action variant.

known bad url no match defanged indicator canonicalisation agent

The canonicalisation variant for parsed URLs.

Compatibility: Threat intelligence feed APIs, URL parsing and canonicalisation libraries, indicator handling in triage agents.

Edge cases to check:

• Keep the defanged original in the case record for safe display, and use the refanged form only for lookups and detonation in controlled tooling.
• Beware over normalising: stripping paths or query strings can merge distinct malicious URLs into one benign looking host.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Y1rHG9hsyj2W4HqaWRk7cA