agent timed out waiting for gobuster recursive scan; the scan failed on large authorized site

TL;DR

Unbounded recursion is what kills the run: every new directory multiplies the work. Cap the recursion depth, enumerate one path prefix at a time with its own output file, and checkpoint each completed prefix. A timeout then costs one prefix, not the whole site.

agent timed out waiting for gobuster recursive scan; the scan failed on large authorized site

Use this when

• gobuster dir with recursion against a large authorized site times out with no usable output.
• The agent re-runs the full recursive scan after each failure.
• The site has deep directory trees that make recursion explode combinatorially.

Not for this skill when

• gobuster cannot connect to the target at all. That is a connectivity problem.
• The site is small and the scan finishes in minutes. Recursion depth is not the issue.
• The engagement scope limits enumeration depth. Respect the scope.

Steps

1. Set an explicit recursion depth limit instead of the default open-ended recursion, and start shallow (depth 2). Verify: the run finishes the shallow pass before attempting anything deeper.
2. Split the wordlist run by top-level path: enumerate /api, /static, /admin and other prefixes as separate invocations, each writing its own output file. Verify: each prefix command runs standalone.
3. Run each prefix under its own timeout, for example timeout 900 gobuster dir -u TARGET_URL/api -w WORDLIST -o api_dirs.txt. Verify: a stuck prefix exits with code 124 and leaves other prefixes intact.
4. Record completed prefixes in a progress file; on retry, run only the missing ones. Verify: the retry plan references the progress file, not the full site.
5. Only deepen recursion on prefixes that actually returned directories. Verify: the deep pass targets a short list of real directories, not the whole site.

Variant phrasings

gobuster recursion exploded on large site

Same depth-cap fix.

agent recursive dir scan never finishes

Symptom phrasing.

gobuster timeout on big authorized target

Target-size variant.

Compatibility: gobuster 3.x. Depth control and per-run output files are standard flags.

Why it happens

Recursive enumeration is multiplicative: each discovered directory spawns a full new wordlist run beneath it. On a large site that is thousands of runs hiding behind one command, and the agent's single timeout covers all of them. Prefix chunking plus depth caps turns one unbounded job into a set of bounded ones the agent can actually schedule and resume.

Edge cases / pitfalls

• Depth 1 misses nested admin panels; depth 5 on a big site never finishes. Start shallow, deepen only where the shallow pass found something.
• Wildcard responses poison recursive runs with false directories. Calibrate gobuster's wildcard handling on a known path before the big run.
• Every discovered directory is a potential rabbit hole. Time-box the deep pass per directory so one huge subtree cannot eat the whole engagement.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VitD2DHcPvvJZir6YiVZsQ