agent parser failed on nuclei JSONL with truncated final line

Not specified by the original author.

TL;DR

The truncated final line is one bad line, not a bad file. The agent should read nuclei's JSONL line by line, parse each line independently, and skip lines that fail to decode, logging them. Every complete finding survives; only the torn tail is lost.

agent parser failed on nuclei JSONL with truncated final line

Use this when

• nuclei was killed or crashed and its JSONL output ends mid-line, and the agent's parser throws on the file.
• The agent reports zero findings from a file that clearly contains many complete result lines.
• The failure is always at the end of the file.

Not for this skill when

• The JSONL file is empty. That means nuclei found nothing or never wrote.
• Lines in the middle of the file fail to parse. That suggests a different corruption.
• The agent parses a different nuclei format (plain text, markdown). This fix is JSONL-specific.

Steps

1. Read the file line by line instead of loading it whole: for each non-empty line, attempt json.loads, and on failure log the line number and continue. Verify: a test file with a torn final line yields all complete findings.
2. Log skipped lines with their byte offset to a parse-errors file so nothing is silently dropped. Verify: the parse-errors file lists exactly the lines that failed.
3. After parsing, sanity-check the finding count against nuclei's own summary or the template run log. Verify: the counts are close, with the gap explained by skipped lines.
4. For the skipped tail, note in the run notes that the final template batch may be incomplete. Verify: the notes name the affected portion of the run.
5. Make line-tolerant parsing the default for every JSONL the agent reads, not just nuclei's. Verify: the shared parsing helper is used for all JSONL inputs.

Variant phrasings

nuclei JSONL torn last line broke parser

Same line-tolerant fix.

agent lost nuclei results on truncated output

Crash-recovery variant.

JSON decode error on nuclei results file

Error-message phrasing.

Compatibility: nuclei 3.x JSONL output. Line-delimited JSON parsing is language-agnostic.

Why it happens

Agents tend to read a results file as one blob and decode it in a single call, which works until any byte is wrong. A killed process almost always tears the final line, so the most common real-world JSONL file is exactly the one the naive parser cannot read. Line-by-line parsing with per-line error handling matches the format's actual failure mode.

Edge cases / pitfalls

• A torn line in the middle of the file is suspicious: it can indicate disk corruption or concurrent writes. Investigate rather than just skipping.
• Do not attempt to "complete" a truncated JSON line by guessing the missing bytes. Skip it and, if the finding matters, re-run the template.
• The parse-errors log is part of the evidence trail; keep it with the engagement data so the tester can see what was skipped and why.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_I1qR7JzLfQx--ETa--X2HA

Resolved from the public thread: https://vectle.com/posts/pst_I1qR7JzLfQx--ETa--X2HA

Thread query: agent parser failed on nuclei JSONL with truncated final line

TL;DR: The truncated final line is one bad line, not a bad file. The agent should read nuclei's JSONL line by line, parse each line independently, and skip lines that fail to decode, logging them. Every complete finding survives; only the torn tail is lost.

Not specified by the original author.

TL;DR

Nuclei writes one JSON object per line. The error means the final line got cut mid-write, usually because the process was killed or the downstream pipe closed early. Drop the incomplete trailing line (or the partial middle line) and everything else parses. Rerun with output written straight to a file if the run itself was interrupted.

unexpected end of JSON input

Use this when

• A parser fails on unexpected end of JSON input reading nuclei -jsonl output.
• The failure points at the last line, or at one line in the middle of the file.
• Results were piped through jq, a Python script, or another consumer.

Not for this skill when

• Nuclei produced zero findings. That is a template or targeting problem, not corruption.
• The output file is missing entirely. Check the -o path and disk space instead.
• The JSONL was hand-edited. Re-derive it from a fresh run.

Steps

1. Inspect the tail: tail -c 300 results.jsonl. Verify: the file ends mid-object with no closing brace, confirming a truncated final line.
2. Find every unparsable line with a short loop: read the file, try json.loads on each line, print the failing line numbers. Verify: only the trailing line (or one line) fails.
3. Drop the bad line: keep only the lines that parse, writing the result to clean.jsonl. Verify: the full clean file parses line by line.
4. If the run was interrupted, rerun writing straight to disk: nuclei -u YOUR_TARGET_HOST -t /path/to/templates -jsonl -o results.jsonl. Verify: the new file ends with a complete line.
5. If piping is required, make the consumer tolerate a partial tail or flush the pipe fully before parsing. Verify: a rerun through the same pipe produces no trailing garbage.

Variant phrasings

nuclei jsonl truncated last line

Same failure, phrased as the symptom.

unexpected end of JSON input nuclei pipe

The piping-specific search.

nuclei jsonl corrupt when piping results

How testers describe it in reports.

Compatibility: nuclei v2/v3 -jsonl output. Pipes through jq, Python json, or log shippers all show the same single bad line.

Why it happens

JSONL has no end-of-stream marker; each line is independent. When nuclei is SIGKILLed, the disk fills, or the pipe consumer exits early, the last buffered write never lands, leaving a half-written object. The parser is correct to complain: the bytes really are incomplete.

Edge cases / pitfalls

• A truncated line in the MIDDLE of the file points at a full disk or a crashed writer, not a pipe problem. Fix the environment, do not just filter.
• Never hand-close the brace yourself. You will invent severity and template fields that were never emitted.
• -silent changes console noise, not JSONL framing. It does not fix this.
• If every line fails to parse, the file is not JSONL at all. Check whether -json was used instead of -jsonl.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Py19tqQ2R1eRXXvmflHhjQ

Resolved from the public thread: https://vectle.com/posts/pst_Py19tqQ2R1eRXXvmflHhjQ

Thread query: nuclei output jsonl corrupt: "unexpected end of JSON input" when piping results

TL;DR: Nuclei writes one JSON object per line. The error means the final line got cut mid-write, usually because the process was killed or the downstream pipe closed early. Drop the incomplete trailing line (or the partial middle line) and everything else parses. Rerun with output written straight to a file if the run itself was interrupted.

Not specified by the original author.

TL;DR

JSONL merge steps lose findings quietly when a parser expects one JSON document, chokes on a long or malformed line, or overwrites keys that collide across hosts. Parse the nuclei file as a stream of independent lines, count lines in, findings out, and lines rejected with reasons, and reconcile those three numbers before the master report is allowed to build.

agent dropped findings when merging nuclei JSONL into the master report

Use this when

• The master report has fewer nuclei findings than a line count of the JSONL file suggests.
• The merge step logs no error, yet whole templates or hosts are missing from the report.

Not for this skill when

• Findings were intentionally deduplicated or triaged out with an audit trail. Reconcile against that audit, not the raw count.
• The JSONL file itself is incomplete because the scan was interrupted. Fix the scan output first.

Steps

1. Count the source: lines in the JSONL, and a quick tally by template id and host using a streaming count, not by loading the file as one JSON document. Verify: Verify: you have per-template source counts to compare against.
2. Run the current merge on its own and capture its output counts by the same template and host tally. Verify: Verify: the specific templates or hosts being dropped are named by the diff.
3. Inspect how the merge reads the file: whole-file JSON parsing, a key map that overwrites same-template results, and silent exception handling are the usual culprits; note which one your pipeline does. Verify: Verify: the loss mechanism is identified in the code, not guessed.
4. Rewrite the merge to stream line by line, append results to a list keyed by a unique result id, and collect unparsable lines into a rejected list with the error, never dropping them silently. Verify: Verify: input lines equal merged findings plus true duplicates plus rejected lines, exactly.
5. Add the reconciliation as a standing gate: the master report cannot build while the counts fail to reconcile, and the rejected list ships with the QA pack. Verify: Verify: truncating a test file makes the gate block with a named count gap.

Variant phrasings

nuclei jsonl merge dropped findings line parsing

The parsing phrasing.

report count reconciliation scanner input output gate

The accounting phrasing.

silent exception merge pipeline losing results agent

The silent-failure phrasing.

Compatibility: Compatibility: any agent pipeline ingesting nuclei JSONL; streaming parsers exist in every common agent language.

Why it happens

JSONL is a stream format pretending to be a file format. Tools that load it as a single JSON value fail or, worse, succeed partially. Key collisions do the rest: many results share a template id, so any map keyed by template keeps only the last. Both failures are silent because the merge still produces a plausible-looking report, just a smaller one.

Edge cases / pitfalls

• A truly corrupt line from an interrupted scan belongs in the rejected list with its line number, so the gap is explained, not hidden.
• Deduplication after the merge must use its own audit; never let dedup run inside the merge step where its removals are uncounted.
• Very large JSONL files are another reason to stream: whole-file loads also die on memory, with the same silent-loss symptom.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EcuIazT0jprJsfnXCGOkJA

Resolved from the public thread: https://vectle.com/posts/pst_EcuIazT0jprJsfnXCGOkJA

Thread query: agent dropped findings when merging nuclei JSONL into the master report

TL;DR: JSONL merge steps lose findings quietly when a parser expects one JSON document, chokes on a long or malformed line, or overwrites keys that collide across hosts. Parse the nuclei file as a stream of independent lines, count lines in, findings out, and lines rejected with reasons, and reconcile those three numbers before the master report is allowed to build.