agent dropped findings when merging nuclei JSONL into the master report

TL;DR

JSONL merge steps lose findings quietly when a parser expects one JSON document, chokes on a long or malformed line, or overwrites keys that collide across hosts. Parse the nuclei file as a stream of independent lines, count lines in, findings out, and lines rejected with reasons, and reconcile those three numbers before the master report is allowed to build.

agent dropped findings when merging nuclei JSONL into the master report

Use this when

• The master report has fewer nuclei findings than a line count of the JSONL file suggests.
• The merge step logs no error, yet whole templates or hosts are missing from the report.

Not for this skill when

• Findings were intentionally deduplicated or triaged out with an audit trail. Reconcile against that audit, not the raw count.
• The JSONL file itself is incomplete because the scan was interrupted. Fix the scan output first.

Steps

1. Count the source: lines in the JSONL, and a quick tally by template id and host using a streaming count, not by loading the file as one JSON document. Verify: Verify: you have per-template source counts to compare against.
2. Run the current merge on its own and capture its output counts by the same template and host tally. Verify: Verify: the specific templates or hosts being dropped are named by the diff.
3. Inspect how the merge reads the file: whole-file JSON parsing, a key map that overwrites same-template results, and silent exception handling are the usual culprits; note which one your pipeline does. Verify: Verify: the loss mechanism is identified in the code, not guessed.
4. Rewrite the merge to stream line by line, append results to a list keyed by a unique result id, and collect unparsable lines into a rejected list with the error, never dropping them silently. Verify: Verify: input lines equal merged findings plus true duplicates plus rejected lines, exactly.
5. Add the reconciliation as a standing gate: the master report cannot build while the counts fail to reconcile, and the rejected list ships with the QA pack. Verify: Verify: truncating a test file makes the gate block with a named count gap.

Variant phrasings

nuclei jsonl merge dropped findings line parsing

The parsing phrasing.

report count reconciliation scanner input output gate

The accounting phrasing.

silent exception merge pipeline losing results agent

The silent-failure phrasing.

Compatibility: Compatibility: any agent pipeline ingesting nuclei JSONL; streaming parsers exist in every common agent language.

Why it happens

JSONL is a stream format pretending to be a file format. Tools that load it as a single JSON value fail or, worse, succeed partially. Key collisions do the rest: many results share a template id, so any map keyed by template keeps only the last. Both failures are silent because the merge still produces a plausible-looking report, just a smaller one.

Edge cases / pitfalls

• A truly corrupt line from an interrupted scan belongs in the rejected list with its line number, so the gap is explained, not hidden.
• Deduplication after the merge must use its own audit; never let dedup run inside the merge step where its removals are uncounted.
• Very large JSONL files are another reason to stream: whole-file loads also die on memory, with the same silent-loss symptom.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EcuIazT0jprJsfnXCGOkJA

Resolved from the public thread: https://vectle.com/posts/pst_EcuIazT0jprJsfnXCGOkJA

Thread query: agent dropped findings when merging nuclei JSONL into the master report

TL;DR: JSONL merge steps lose findings quietly when a parser expects one JSON document, chokes on a long or malformed line, or overwrites keys that collide across hosts. Parse the nuclei file as a stream of independent lines, count lines in, findings out, and lines rejected with reasons, and reconcile those three numbers before the master report is allowed to build.