agent's regex-based url extractor missed shortened links; parsing failed on t.co wraps

TL;DR

A regex tuned on plain http links misses the modern message: links wrapped by trackers, shortened by social platforms, split across lines, or hidden in HTML href attributes with entities. Replace pattern-only extraction with structured parsing: parse the HTML for href targets, decode entities, unwrap known redirectors where safe, and keep a maintained short-domain list. Validate every candidate by parsing it as a URL, not by trusting the regex shape.

agent's regex-based url extractor missed shortened links; parsing failed on t.co wraps

Use this when

• Extraction output omits links that are visibly in the message, especially wrapped ones.
• Shortened links reach verdict steps as unexpanded short forms or not at all.

Not for this skill when

• Links are present only inside attachments. That is attachment extraction, a different stage.
• The message is plain text with standard links that extract fine. The issue is elsewhere.

Steps

1. Build a test set from real campaign messages covering wrapped, shortened, entity-encoded and line-broken links. Verify: the misses are reproduced against the current extractor.
2. Add HTML href extraction from the parsed body alongside text regex, merging and deduplicating results. Verify: href-only links now appear.
3. Decode HTML entities and join line-broken URLs before validation. Verify: encoded samples extract to their true form.
4. Maintain a shortener and wrapper list whose links are flagged for expansion in the verdict step. Verify: t.co and major wrappers are flagged, not dropped.
5. Run the test set plus a regression set of normal messages. Verify: extraction recall is complete on the test set with no new false URLs.

Variant phrasings

url extractor misses shortened links t.co regex

The short link variant.

extract href links html email agent parsing

The href parsing variant.

Compatibility: Agent message parsing in Python and similar stacks, major link wrapping services.

Why it happens

Regex extraction encodes the author idea of a link from the day it was written. Senders, platforms and attackers all keep inventing new link shapes, so extraction rots steadily until a campaign arrives wearing the shape nobody added.

Edge cases / pitfalls

• Over-extraction is also a failure: tracking pixels and unsubscribe links flood verdict steps. Tag link roles where you can.
• Expansion of shortened links must happen in a safe resolver, since the short link is itself untrusted.
• Keep the raw extracted string with the normalised one; discrepancies between them are sometimes the finding.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2FgOt-oFGxEtNvQL8QkvZQ