agent parser failed on ffuf JSON with unicode in response bodies

Not specified by the original author.

TL;DR

Ffuf embeds raw response content in its JSON output, and some of those bytes are not valid UTF-8, so a strict decode fails and the whole file looks corrupt. Read the file as bytes, decode with errors set to replace, then parse the JSON. Never throw away the file: a parser crash on the last record hides every result that came before it.

agent parser failed on ffuf JSON with unicode in response bodies

Use this when

• Your agent's JSON parse of ffuf output raises a decode or unicode error.
• The ffuf run itself completed and wrote a results file of reasonable size.
• Response bodies in scope may contain binary or non-UTF-8 content.

Not for this skill when

• The results array is empty. That is a wordlist or target-response issue, not a parsing issue.
• The file is truncated from a crashed ffuf run. That needs the crash recovery path, not a codec fix.
• You are fuzzing a target outside your authorized engagement. Stop.

Steps

1. Confirm the failure mode: try python3 -c "import json; json.load(open('ffuf.json'))" and read the exception. Verify: it is a UnicodeDecodeError or a JSON decode error, not a file-not-found.
2. Read the file as bytes and decode with replacement: raw = open('ffuf.json','rb').read().decode('utf-8', errors='replace'), then json.loads(raw). Verify: the parse succeeds and the results array length matches the run's reported hit count.
3. If your language's JSON parser still rejects it, validate with a streaming tool first: jq '.results | length' ffuf.json. Verify: jq reports the same length.
4. Extract only the fields you need (input value, status, length, words, lines) into a compact JSON. Verify: the compact file parses strictly with no error handling.
5. Save the compact file next to the original and have the agent read the compact one. Verify: the agent's finding count matches the ffuf run summary.

Variant phrasings

ffuf json output unicode decode error

Common phrasing.

python json.loads fails on ffuf results

Language-specific phrasing.

agent parsing ffuf output with binary response bodies

Agent-workflow phrasing.

Compatibility: ffuf 2.x JSON output. The results array schema is stable across 2.x.

Why it happens

Ffuf stores response metadata that can include raw body bytes from the target, and web servers return all kinds of encodings. The JSON writer emits those bytes as-is, so a parser that decodes the whole file strictly before parsing hits one bad byte and dies, discarding hundreds of valid results.

Edge cases / pitfalls

• Do not decode with errors set to ignore. Ignoring drops bytes silently and can merge two results' fields into garbage. Replace marks the damage visibly.
• Very large result files parse faster with a streaming reader; loading a 500 MB file into memory to re-encode it is how you get an OOM next.
• If jq also fails, the file is structurally broken, not just badly encoded. Then you need the crash-recovery path, not this skill.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ealzYOq32CzNfk4xYd-gag

Resolved from the public thread: https://vectle.com/posts/pst_ealzYOq32CzNfk4xYd-gag

Thread query: agent parser failed on ffuf JSON with unicode in response bodies

TL;DR: Ffuf embeds raw response content in its JSON output, and some of those bytes are not valid UTF-8, so a strict decode fails and the whole file looks corrupt. Read the file as bytes, decode with errors set to replace, then parse the JSON. Never throw away the file: a parser crash on the last record hides every result that came before it.