agent hallucinated open directories from failed gobuster run on authorized target

TL;DR

Every reported directory needs a receipt: the exact line in gobuster's raw output that proves it. The agent should extract findings only from the tool's output file and refuse to report anything it cannot point to. A failed run produces no findings, full stop.

agent hallucinated open directories from failed gobuster run on authorized target

Use this when

• The agent's findings list includes directories that do not appear anywhere in gobuster's output.
• The gobuster run failed (timeout, crash, connection error) yet the agent still reported "discoveries".
• A human tester cannot reproduce a reported directory from the evidence.

Not for this skill when

• gobuster returned wildcard false positives. Those are real tool output needing wildcard calibration, not hallucinations.
• The directories are real but came from a different tool's output. That is a sourcing mix-up.
• The run succeeded and the findings match the output. Then the pipeline is healthy.

Steps

1. Change the extraction rule: findings come only from parsing gobuster's output file, never from the agent's summary of what the run "probably" found. Verify: each finding has a matching raw output line.
2. Before extracting, check the run's exit status and output integrity: a failed or truncated run yields zero findings by policy. Verify: a killed run's output file produces an empty findings list.
3. Attach the evidence line to each finding in the notes (path, status code, size as gobuster reported it). Verify: every finding in the notes carries its source line.
4. Add a verification pass: for each reported directory, confirm it appears verbatim in the raw output via a text search. Verify: the verification log shows one match per finding.
5. If a finding fails verification, drop it and log why, then investigate whether the agent is summarizing instead of parsing. Verify: the dropped-finding log exists and is non-empty when hallucinations occurred.

Variant phrasings

agent invented gobuster findings

Same evidence-required fix.

directories reported from failed enumeration run

Failed-run variant.

findings with no tool output to back them

Evidence-focused phrasing.

Compatibility: gobuster 3.x output formats. The evidence rule is harness-agnostic and applies to any enumeration tool.

Why it happens

Language models are prediction machines: given a failed run and pressure to produce findings, they generate plausible-looking directories from the target's naming patterns instead of reporting nothing. The agent is not lying; it is completing the pattern. An evidence-required extraction rule removes the opportunity by making "no output" mean "no findings" mechanically, not judgmentally.

Edge cases / pitfalls

• "No findings" is a valid, reportable outcome. Agents that feel punished for empty results hallucinate more; make empty an acceptable answer in the runbook.
• Wildcard responses are the honest version of this problem: gobuster really did return those paths. Calibrate wildcard handling separately; do not confuse the two.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstJ2tfRt2O9zwQzvy2PT0kA