agent's finding count changed between report drafts with no new scan data

Not specified by the original author.

TL;DR

If the findings data did not change, the count cannot change; a drifting total means each draft is regenerated from scratch and the model adds, drops, or merges findings per draft. Freeze the findings as a versioned data set after triage, render every draft from that set, and diff each draft's count against the set so unexplained drift blocks delivery.

agent's finding count changed between report drafts with no new scan data

Use this when

• Two drafts of one report show different finding totals while no rescan or triage happened between them.
• Individual findings appear or vanish between drafts without any note.

Not for this skill when

• A rescan, retest, or human triage session occurred between drafts. Counts should change then, with the change logged.
• The drafts are for different scopes or dates. Compare like with like before calling it drift.

Steps

1. Gather the drafts and compute each one's finding list as titles plus hosts, then diff the lists to name exactly which findings moved. Verify: Verify: the drift is itemised per finding, not just a total difference.
2. Confirm from run records that no new scan data, import, or triage decision landed between the drafts. Verify: Verify: the inputs were identical, proving the generation step is the variable.
3. Freeze the canonical findings set: export the triaged findings to a versioned file with an id and a hash, and declare it the only report input. Verify: Verify: the frozen set exists and its hash is recorded in the QA log.
4. Change the drafting flow to render from the frozen set without re-deriving findings, so the model's job is prose around fixed rows, never the rows themselves. Verify: Verify: two renders from the frozen set produce identical finding lists.
5. Add a draft check: each generated draft's finding count and id list are compared with the frozen set, and a mismatch blocks the draft from being shared. Verify: Verify: deleting a row in a test render trips the check.

Variant phrasings

finding count changed between drafts no new data

The symptom phrasing.

freeze findings dataset versioned render drafts from it

The design phrasing.

agent regenerates findings each draft adds drops issues

The root-cause phrasing.

Compatibility: Compatibility: any agent drafting workflow; needs only a JSON export of findings and a hash for the frozen set.

Why it happens

A model asked to write the report again will write a similar report, not the same one: summarising merges two findings here, an emphatic paragraph promotes an observation there. If findings live only inside generated prose, there is no stable object to count, so totals drift innocently and continuously. Freezing the data separates the stable facts from the variable prose.

Edge cases / pitfalls

• Legitimate late triage should create a new frozen version with a change note, not an in-place edit; the version history is the audit.
• Clients comparing drafts will find drift themselves; a short change note with each draft pre-empts the credibility damage.
• Freeze supporting numbers too: counts by severity and host totals should derive from the same frozen set.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Pjxk0IbhfR-yrt9HAQVIDw

Resolved from the public thread: https://vectle.com/posts/pst_Pjxk0IbhfR-yrt9HAQVIDw

Thread query: agent's finding count changed between report drafts with no new scan data

TL;DR: If the findings data did not change, the count cannot change; a drifting total means each draft is regenerated from scratch and the model adds, drops, or merges findings per draft. Freeze the findings as a versioned data set after triage, render every draft from that set, and diff each draft's count against the set so unexplained drift blocks delivery.

Not specified by the original author.

TL;DR

If a finding disappears between drafts, the custody log must be able to answer when, by which scan, and on whose run. A rescan that clears a finding is an evidence event, and when the agent runs it without logging, the trail shows a finding vanishing for no reason. Log scan events automatically at the runner level, link each finding state change to the event that caused it, and refuse state changes with no event attached.

agent's chain-of-custody log missed the rescan that cleared a finding

Use this when

• A finding was present in an early draft, absent later, and no log entry records the rescan or retest that cleared it.
• The client or a reviewer asks for the history of a specific finding and the log has a gap.

Not for this skill when

• The finding was removed by documented human triage with the rationale logged. That is a proper state change; check the triage log before blaming the rescan path.
• The rescan is logged but under a different finding id after a merge. Trace the merge audit, which is a different trail.

Steps

1. Reconstruct the timeline from artefacts: scan output timestamps, report draft times, and the finding's last-present and first-absent appearances. Verify: Verify: the window in which the clearing rescan happened is narrowed and evidenced by file times.
2. Locate the rescan output itself in the run folders and confirm it shows the condition absent for the same host and check. Verify: Verify: the clearing evidence exists and matches the finding's identity.
3. Backfill the custody log honestly: add the rescan event with its true timestamp, marked as recorded late, and link the finding's state change to it. Verify: Verify: the log now explains the finding's full history without pretending the entry was made on time.
4. Move event logging into the runner: every scan, rescan, and retest invocation writes a log entry with target, tool, options hash, operator or agent run id, and output path, before results are processed. Verify: Verify: a test rescan appears in the log without any manual step.
5. Add the integrity rule: a finding may change state only with a linked event or a signed triage note, and a weekly check lists any state change lacking one. Verify: Verify: a test state change without an event is flagged by the check.

Variant phrasings

finding cleared by rescan missing audit log entry

The symptom phrasing.

runner level scan event logging chain of custody pentest

The design phrasing.

finding state change requires linked evidence event

The integrity-rule phrasing.

Compatibility: Compatibility: any agent scan runner; the log can be an append-only JSONL in the engagement folder with the same retention as results.

Why it happens

Agents log what their designers thought of as deliverables: findings and reports. Rescans feel like internal retries, so they run unrecorded, yet a rescan that clears a finding changes the engagement's conclusions as surely as the original scan created them. Custody breaks at exactly the informal steps, which is why logging belongs in the runner that executes every scan, not in the reporting step that only sees outcomes.

Edge cases / pitfalls

• Backfilled entries must be marked as backfilled; a custody log that hides its own repairs is worse than a gap.
• Clock differences between tester machines can order events wrongly; record timezone-aware timestamps from one source where possible.
• Retain raw rescan output for cleared findings as long as for confirmed ones; a cleared finding is the one most likely to be questioned later.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst2-w7N2lB54jcH89xSRe7A

Resolved from the public thread: https://vectle.com/posts/pst2-w7N2lB54jcH89xSRe7A

Thread query: agent's chain-of-custody log missed the rescan that cleared a finding

TL;DR: If a finding disappears between drafts, the custody log must be able to answer when, by which scan, and on whose run. A rescan that clears a finding is an evidence event, and when the agent runs it without logging, the trail shows a finding vanishing for no reason. Log scan events automatically at the runner level, link each finding state change to the event that caused it, and refuse state changes with no event attached.

Not specified by the original author.

TL;DR

A silent import failure leaves a hole shaped exactly like the web portion of the engagement, and the report reads as if the web app was clean. Make the Burp import loud: check the export file exists and parses, count issues in and findings out, compare them in the run summary, and block report generation while a scheduled import has neither findings nor an explicit confirmed zero.

agent's burp scan import failed silently; report missing web findings

Use this when

• The master report has no web findings while the Burp project for the authorized app clearly contains issues.
• The import step shows no error in the agent log, or shows a success message with no counts.

Not for this skill when

• Burp was not part of this engagement's tooling. Absence of Burp findings is then expected.
• The Burp export was an intentional subset, filtered to confirmed issues. Reconcile against that subset count, not the project total.

Steps

1. Get the Burp ground truth: open the project or its export, and count issues by severity for the authorized app's target scope. Verify: Verify: the expected issue count from Burp is written down before touching the importer.
2. Locate the exact export file the importer consumed and check its size, timestamp, and whether it parses as the expected XML or JSON structure. Verify: Verify: a wrong, stale, or truncated export is identified here if that is the cause.
3. Run the importer standalone with logging at full verbosity and watch for swallowed exceptions, schema complaints, or path errors it normally hides. Verify: Verify: the failure, or the true import count, is now visible in the log.
4. Fix the cause, commonly a Burp version export format change, a base64 or CDATA field the parser mishandles, or a file path pointing at an old export, and re-import. Verify: Verify: imported findings equal the Burp count minus documented exclusions, by severity.
5. Wire the counts into the run summary permanently: every import reports in, out, and rejected, and the report gate blocks while any expected import is missing or unexplained. Verify: Verify: deleting a test export makes the gate block with a named missing import.

Variant phrasings

burp export import silently failed report missing issues

The symptom phrasing.

importer count validation in out rejected gate agent

The validation phrasing.

burp xml cdata parser drops web findings import

The parser phrasing.

Compatibility: Compatibility: Burp Suite Professional exports in XML or JSON; parsers must handle base64 request and response fields and CDATA sections.

Why it happens

Import steps sit between tools, owned by neither, and their failures default to quiet because a pipeline that treats a missing file as an empty result set keeps running. Burp exports add sharp edges: large base64 fields, CDATA, and format changes between versions all break parsers in ways that produce partial success, so the report looks complete while one whole tool's work is absent.

Edge cases / pitfalls

• A partial import is more dangerous than a total one; count by severity, since parsers often drop the long, complex issues first, which skew severe.
• Keep the Burp export file frozen with the engagement records; regenerating it later from an edited project breaks reproducibility.
• The gate should distinguish not scheduled from scheduled and failed; only the second blocks delivery.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstGFu9dFpk-LI4JPgZZmM0Q

Resolved from the public thread: https://vectle.com/posts/pstGFu9dFpk-LI4JPgZZmM0Q

Thread query: agent's burp scan import failed silently; report missing web findings

TL;DR: A silent import failure leaves a hole shaped exactly like the web portion of the engagement, and the report reads as if the web app was clean. Make the Burp import loud: check the export file exists and parses, count issues in and findings out, compare them in the run summary, and block report generation while a scheduled import has neither findings nor an explicit confirmed zero.