recon agent hit RDAP rate limit mid-engagement; lookups failed on authorized targets

TL;DR

RDAP servers throttle aggressively, so the agent should act like every lookup costs money: cache every response on disk, deduplicate domains by registrar before querying, and back off when throttled. Most engagements need a fraction of the lookups the agent actually makes.

recon agent hit RDAP rate limit mid-engagement; lookups failed on authorized targets

Use this when

• RDAP lookups for authorized target domains start returning 429s or timeouts partway through enumeration.
• The agent looks up the same domain or IP block multiple times across the run.
• The failure cascade starts at the lookup stage and stalls everything downstream.

Not for this skill when

• The RDAP server refuses the queries entirely (403, invalid responses). That is an access or format problem.
• The engagement has no domain or infrastructure enumeration phase. Then RDAP is irrelevant.
• Lookups succeed but the data is wrong. That is a data-quality problem.

Steps

1. Add a local cache: before any RDAP query, check a JSON cache file keyed by domain or IP; after a successful query, store the response. Verify: the second run of the same target list makes zero RDAP calls.
2. Deduplicate the target list before querying: group domains by registrar and IPs by announced block, and query each group once. Verify: the deduped list is shorter than the raw list.
3. On a 429, back off exponentially with jitter and resume from the cache-aware queue instead of restarting. Verify: a forced throttle in testing pauses the queue rather than failing it.
4. Pace the queue below the server's documented limits with a fixed delay between calls. Verify: a 15-minute test batch completes without a single 429.
5. Log throttled queries with timestamps in the run notes so the tester can see whether the data is complete. Verify: the notes list any domains that never got a successful lookup.

Variant phrasings

RDAP throttling killed recon lookups

Same cache-and-backoff fix.

agent re-queries RDAP for same domains

Deduplication-focused variant.

WHOIS/RDAP rate limit mid-enumeration

Broader phrasing.

Compatibility: any RDAP client or script. The cache file pattern is client-agnostic.

Why it happens

Recon agents expand targets enthusiastically: every subdomain becomes a lookup, every IP becomes a lookup, and the same infrastructure gets queried once per branch of the enumeration tree. RDAP providers rate-limit exactly this pattern. Without a cache the agent pays full price for every duplicate, hits the wall fast, and the whole downstream pipeline stalls on missing data.

Edge cases / pitfalls

• Cached RDAP data goes stale; for long engagements, expire cache entries after a few days rather than trusting them forever.
• Some registries throttle harder than others. Track per-server throttle counts and pace each server separately.
• A domain that never resolves through RDAP is a data gap, not a dead end; note it and move on rather than retrying it into another rate limit.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_DjsgTwS5ZjNCDJfETBH7Qg