rate limit hit mid-run: agent ZAP spider failed after WAF blocked authorized target scan

TL;DR

A WAF block during an authorized test is a coordination signal, not a puzzle to solve. The agent should detect block responses, stop the spider, and hand the run to the tester to confirm the scanner is allowlisted for the test window, then resume at a slower crawl rate. Evading the WAF is never the fix on an authorized engagement.

rate limit hit mid-run: agent ZAP spider failed after WAF blocked authorized target scan

Use this when

• ZAP's spider starts returning 403s, 429s, or block pages partway through crawling an authorized target.
• The agent keeps spidering through the block, burning the run and risking an IP ban.
• The engagement includes a test window but nobody confirmed the scanner source is allowlisted.

Not for this skill when

• The target never authorized automated scanning. Then the block is the correct outcome; stop entirely.
• ZAP fails before any request goes out. That is a ZAP configuration problem.
• The block comes from the agent's own network egress, not the target. That is an infrastructure issue.

Steps

1. Teach the agent the block signature: a sudden run of 403/429 responses or a known block page body means stop the spider immediately. Verify: a test run against a blocking endpoint halts within a minute of the first block.
2. Save the spider's progress (visited URLs, scan queue) before stopping so the run resumes instead of restarting. Verify: the saved queue file lists unvisited URLs.
3. Hand off to the human tester with the evidence: block timestamps, response codes, and the scanner source address. The tester confirms allowlisting for the test window with the target team. Verify: the tester confirms in writing before the agent resumes.
4. Resume the spider at a reduced crawl rate (fewer concurrent requests, delay between requests) even after allowlisting. Verify: the resumed run completes without new blocks.
5. Log the block window and the allowlist confirmation in the engagement notes. Verify: the report can show exactly when scanning paused and why.

Variant phrasings

ZAP spider blocked by WAF mid-scan

Same coordination fix.

agent spidering tripped target rate limit

Rate-limit phrasing.

WAF blocked authorized ZAP scan

Target-side phrasing.

Compatibility: OWASP ZAP 2.12+. The spider's queue can be persisted via the ZAP API; the coordination step is process, not tooling.

Why it happens

Agents see a spider as a fire-and-forget task and WAF blocks as transient errors to retry through. On an authorized test the block usually means the allowlist step was missed or expired, and retrying through it looks exactly like an attack the WAF is designed to stop. Stopping and coordinating is both the effective fix and the professional one.

Edge cases / pitfalls

• Never rotate IPs, user agents, or request patterns to dodge the block. On an authorized engagement that is indistinguishable from evasion and can terminate the test.
• If the target team cannot allowlist in time, the agent rescopes to what is reachable and documents the gap; it does not find another way in.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mt0PoLtOtoFr58hQDMlaCA