## TL;DR
A regex tuned on plain http links misses the modern message: links wrapped by trackers, shortened by social platforms, split across lines, or hidden in HTML href attributes with entities. Replace pattern-only extraction with structured parsing: parse the HTML for href targets, decode entities, unwrap known redirectors where safe, and keep a maintained short-domain list. Validate every candidate by parsing it as a URL, not by trusting the regex shape.

```text
agent's regex-based url extractor missed shortened links; parsing failed on t.co wraps
```

## Use this when
- Extraction output omits links that are visibly in the message, especially wrapped ones.
- Shortened links reach verdict steps as unexpanded short forms or not at all.

## Not for this skill when
- Links are present only inside attachments. That is attachment extraction, a different stage.
- The message is plain text with standard links that extract fine. The issue is elsewhere.

## Steps
1. Build a test set from real campaign messages covering wrapped, shortened, entity-encoded and line-broken links. Verify: the misses are reproduced against the current extractor.
2. Add HTML href extraction from the parsed body alongside text regex, merging and deduplicating results. Verify: href-only links now appear.
3. Decode HTML entities and join line-broken URLs before validation. Verify: encoded samples extract to their true form.
4. Maintain a shortener and wrapper list whose links are flagged for expansion in the verdict step. Verify: t.co and major wrappers are flagged, not dropped.
5. Run the test set plus a regression set of normal messages. Verify: extraction recall is complete on the test set with no new false URLs.

## Variant phrasings
### url extractor misses shortened links t.co regex
The short link variant.
### extract href links html email agent parsing
The href parsing variant.

Compatibility: Agent message parsing in Python and similar stacks, major link wrapping services.

## Why it happens
Regex extraction encodes the author idea of a link from the day it was written. Senders, platforms and attackers all keep inventing new link shapes, so extraction rots steadily until a campaign arrives wearing the shape nobody added.

## Edge cases / pitfalls
- Over-extraction is also a failure: tracking pixels and unsubscribe links flood verdict steps. Tag link roles where you can.
- Expansion of shortened links must happen in a safe resolver, since the short link is itself untrusted.
- Keep the raw extracted string with the normalised one; discrepancies between them are sometimes the finding.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2FgOt-oFGxEtNvQL8QkvZQ