## TL;DR
Unbounded recursion is what kills the run: every new directory multiplies the work. Cap the recursion depth, enumerate one path prefix at a time with its own output file, and checkpoint each completed prefix. A timeout then costs one prefix, not the whole site.

```text
agent timed out waiting for gobuster recursive scan; the scan failed on large authorized site
```

## Use this when
- `gobuster dir` with recursion against a large authorized site times out with no usable output.
- The agent re-runs the full recursive scan after each failure.
- The site has deep directory trees that make recursion explode combinatorially.

## Not for this skill when
- gobuster cannot connect to the target at all. That is a connectivity problem.
- The site is small and the scan finishes in minutes. Recursion depth is not the issue.
- The engagement scope limits enumeration depth. Respect the scope.

## Steps
1. Set an explicit recursion depth limit instead of the default open-ended recursion, and start shallow (depth 2). Verify: the run finishes the shallow pass before attempting anything deeper.
2. Split the wordlist run by top-level path: enumerate `/api`, `/static`, `/admin` and other prefixes as separate invocations, each writing its own output file. Verify: each prefix command runs standalone.
3. Run each prefix under its own timeout, for example `timeout 900 gobuster dir -u TARGET_URL/api -w WORDLIST -o api_dirs.txt`. Verify: a stuck prefix exits with code 124 and leaves other prefixes intact.
4. Record completed prefixes in a progress file; on retry, run only the missing ones. Verify: the retry plan references the progress file, not the full site.
5. Only deepen recursion on prefixes that actually returned directories. Verify: the deep pass targets a short list of real directories, not the whole site.

## Variant phrasings
### gobuster recursion exploded on large site
Same depth-cap fix.
### agent recursive dir scan never finishes
Symptom phrasing.
### gobuster timeout on big authorized target
Target-size variant.

Compatibility: gobuster 3.x. Depth control and per-run output files are standard flags.

## Why it happens
Recursive enumeration is multiplicative: each discovered directory spawns a full new wordlist run beneath it. On a large site that is thousands of runs hiding behind one command, and the agent's single timeout covers all of them. Prefix chunking plus depth caps turns one unbounded job into a set of bounded ones the agent can actually schedule and resume.

## Edge cases / pitfalls
- Depth 1 misses nested admin panels; depth 5 on a big site never finishes. Start shallow, deepen only where the shallow pass found something.
- Wildcard responses poison recursive runs with false directories. Calibrate gobuster's wildcard handling on a known path before the big run.
- Every discovered directory is a potential rabbit hole. Time-box the deep pass per directory so one huge subtree cannot eat the whole engagement.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VitD2DHcPvvJZir6YiVZsQ