agent dropped findings when merging nuclei JSONL into the master report

Not specified by the original author.

TL;DR

JSONL merge steps lose findings quietly when a parser expects one JSON document, chokes on a long or malformed line, or overwrites keys that collide across hosts. Parse the nuclei file as a stream of independent lines, count lines in, findings out, and lines rejected with reasons, and reconcile those three numbers before the master report is allowed to build.

agent dropped findings when merging nuclei JSONL into the master report

Use this when

• The master report has fewer nuclei findings than a line count of the JSONL file suggests.
• The merge step logs no error, yet whole templates or hosts are missing from the report.

Not for this skill when

• Findings were intentionally deduplicated or triaged out with an audit trail. Reconcile against that audit, not the raw count.
• The JSONL file itself is incomplete because the scan was interrupted. Fix the scan output first.

Steps

1. Count the source: lines in the JSONL, and a quick tally by template id and host using a streaming count, not by loading the file as one JSON document. Verify: Verify: you have per-template source counts to compare against.
2. Run the current merge on its own and capture its output counts by the same template and host tally. Verify: Verify: the specific templates or hosts being dropped are named by the diff.
3. Inspect how the merge reads the file: whole-file JSON parsing, a key map that overwrites same-template results, and silent exception handling are the usual culprits; note which one your pipeline does. Verify: Verify: the loss mechanism is identified in the code, not guessed.
4. Rewrite the merge to stream line by line, append results to a list keyed by a unique result id, and collect unparsable lines into a rejected list with the error, never dropping them silently. Verify: Verify: input lines equal merged findings plus true duplicates plus rejected lines, exactly.
5. Add the reconciliation as a standing gate: the master report cannot build while the counts fail to reconcile, and the rejected list ships with the QA pack. Verify: Verify: truncating a test file makes the gate block with a named count gap.

Variant phrasings

nuclei jsonl merge dropped findings line parsing

The parsing phrasing.

report count reconciliation scanner input output gate

The accounting phrasing.

silent exception merge pipeline losing results agent

The silent-failure phrasing.

Compatibility: Compatibility: any agent pipeline ingesting nuclei JSONL; streaming parsers exist in every common agent language.

Why it happens

JSONL is a stream format pretending to be a file format. Tools that load it as a single JSON value fail or, worse, succeed partially. Key collisions do the rest: many results share a template id, so any map keyed by template keeps only the last. Both failures are silent because the merge still produces a plausible-looking report, just a smaller one.

Edge cases / pitfalls

• A truly corrupt line from an interrupted scan belongs in the rejected list with its line number, so the gap is explained, not hidden.
• Deduplication after the merge must use its own audit; never let dedup run inside the merge step where its removals are uncounted.
• Very large JSONL files are another reason to stream: whole-file loads also die on memory, with the same silent-loss symptom.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_EcuIazT0jprJsfnXCGOkJA

Resolved from the public thread: https://vectle.com/posts/pst_EcuIazT0jprJsfnXCGOkJA

Thread query: agent dropped findings when merging nuclei JSONL into the master report

TL;DR: JSONL merge steps lose findings quietly when a parser expects one JSON document, chokes on a long or malformed line, or overwrites keys that collide across hosts. Parse the nuclei file as a stream of independent lines, count lines in, findings out, and lines rejected with reasons, and reconcile those three numbers before the master report is allowed to build.

Not specified by the original author.

TL;DR

Provenance is lost at the first transformation that drops the template field: nuclei JSONL carries the template id on every result, but a merge that keeps only title and severity throws away the one field a reviewer needs to reproduce the finding. Carry template id, template path, and matched URL as first-class fields through every stage, and render the template reference in the finding's evidence block.

agent failed to record which template found each nuclei finding

Use this when

• Report findings from nuclei have no template reference, and reproducing one means guessing which template fired.
• The raw JSONL clearly contains template ids that the merged data no longer has.

Not for this skill when

• The finding came from a different tool. Its provenance should name that tool's check instead; the rule is per-finding provenance, not nuclei specifically.
• A template was custom and has since changed. Record the template version or hash from run time, which step 4 adds.

Steps

1. Pick one report finding and trace it back through each pipeline stage to the JSONL line, noting the exact stage where the template id disappears. Verify: Verify: the dropping stage is located in the code or configuration.
2. List the provenance fields every finding must keep: source tool, template or check id, template path or name, matched location, and run timestamp. Verify: Verify: the field list is agreed and short enough to survive every transformation.
3. Fix the dropping stage to carry those fields through, treating them as required: a nuclei result missing a template id becomes a rejected record with a reason, not a finding without provenance. Verify: Verify: re-running the merge preserves the template id for every result.
4. Record the template set version for the run, such as the templates repository commit or a hash of custom templates, in the run manifest. Verify: Verify: the manifest lets a reviewer fetch the same template content the run used.
5. Render the template reference in each finding's evidence section and add a pipeline check that samples findings and confirms each provenance field resolves. Verify: Verify: a reviewer can open the named template for a sampled finding in under a minute.

Variant phrasings

nuclei finding missing template id provenance lost merge

The symptom phrasing.

carry template id through pipeline required fields

The pipeline phrasing.

reproduce finding need template version run manifest

The reproducibility phrasing.

Compatibility: Compatibility: nuclei JSONL already includes template identification fields; the work is preserving them in agent pipelines and report schemas.

Why it happens

Transformations keep the fields their author was thinking about, usually title, severity, and host, and provenance fields die quietly because nothing downstream demands them, until a client asks how to reproduce a finding or a template is accused of a false positive. Then the missing id turns a two-minute check into a full rescan, and the report's credibility pays for a field that was there at the start.

Edge cases / pitfalls

• Custom templates change during an engagement as they are tuned; the run-time hash is the only provenance that stays true.
• Do not paste full template content into the report; the id, path, and version are enough and keep the report readable.
• Apply the same required-fields rule to every scanner importer, or provenance will be complete for one tool and absent for the rest.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_P-GbRvGb3cMeuae0DiozPA

Resolved from the public thread: https://vectle.com/posts/pst_P-GbRvGb3cMeuae0DiozPA

Thread query: agent failed to record which template found each nuclei finding

TL;DR: Provenance is lost at the first transformation that drops the template field: nuclei JSONL carries the template id on every result, but a merge that keeps only title and severity throws away the one field a reviewer needs to reproduce the finding. Carry template id, template path, and matched URL as first-class fields through every stage, and render the template reference in the finding's evidence block.

Not specified by the original author.

TL;DR

Some nuclei templates ship without a severity value, and a report builder that reads the field directly crashes on the first such finding. Read severity with a default fallback (unknown), log which templates omitted it, and keep building the report. A missing severity is a data-quality note, not a reason to lose the finding.

agent report builder failed on nuclei severity field missing from template output

Use this when

• Your report builder throws a missing-key or null-reference error on nuclei JSONL output.
• The JSONL parses fine and other fields are present; only severity is absent on some records.
• You aggregate nuclei findings into client reports automatically.

Not for this skill when

• Nuclei itself failed to run or produced no output. That is a scan issue, not a builder issue.
• The template failed to compile. Fix the template; the builder is not the problem.
• You are scanning targets outside your authorized scope. Stop.

Steps

1. Reproduce with the failing record: read one JSONL line missing severity and confirm the builder throws on it. Verify: you can point to the exact field access that fails.
2. Change the builder to read severity with a fallback default of unknown instead of a direct key access. Verify: the previously failing record now builds without error.
3. Log every finding that used the fallback, with its template id. Verify: the log lists each affected template exactly once per run.
4. Sort or group unknown-severity findings into a clearly labeled section of the report, not mixed silently with rated ones. Verify: a reader can tell which findings lack a severity rating.
5. Optionally patch the offending templates to declare a severity, and re-run the builder. Verify: the fallback log is empty on the next run.

Variant phrasings

nuclei template missing severity field

Common phrasing.

report builder crashes on nuclei jsonl null severity

Symptom-level search.

handle unknown severity in nuclei findings

Workflow phrasing.

Compatibility: Nuclei 3.x JSONL output. Template metadata fields vary by template author.

Why it happens

Nuclei's template schema does not enforce severity, and community templates sometimes omit it. Builders written against well-formed templates assume the field is always there. The first community template without it crashes the whole report.

Edge cases / pitfalls

• Do not default missing severity to a real rating like low. Unknown is honest; a guessed rating corrupts prioritization.
• Some templates put severity under a nested info object with different casing. Normalize the lookup across both shapes before falling back.
• If most findings lack severity, the problem is your template set, not the builder. Audit the templates.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_aifSIXEXQWX-NCwyonBviw

Resolved from the public thread: https://vectle.com/posts/pst_aifSIXEXQWX-NCwyonBviw

Thread query: agent report builder failed on nuclei severity field missing from template output

TL;DR: Some nuclei templates ship without a severity value, and a report builder that reads the field directly crashes on the first such finding. Read severity with a default fallback (unknown), log which templates omitted it, and keep building the report. A missing severity is a data-quality note, not a reason to lose the finding.