the agent's exploitability verdict relied on the LLM reading advisor

Not specified by the original author.

TL;DR: Stop letting the model vote on reachability from vibes - require it to produce a call path from an entry point to the vulnerable function, and default to "unknown, treat as reachable" when it cannot. Pin the advisory text it sees and the tool versions it calls, so identical inputs can only produce identical verdicts.

the agent's exploitability assessment contradicted itself between runs - the LLM couldn't decide if the vulnerable function was reachable

Steps

1. Change the verdict schema so reachability only allows three values: "reachable" (with a call path), "not_reachable" (with full-graph evidence of no path), and "unknown." Expected: the model can no longer emit a bare "not reachable" with nothing behind it.

2. Require a structured "call_path" field: an ordered list of caller function names from an entry point down to the vulnerable function. Expected: every verdict ships with the path, or an explicit empty path plus "unknown."

3. Set the default: an empty call path means "unknown," and your policy treats "unknown" as "patch it." Expected: the flip-flop between "reachable" and "not reachable" settles into a stable "unknown" until real evidence exists.

4. Pin the inputs: snapshot the advisory text, SBOM, and code revision into the run artifact and hash them. Expected: two runs on the same hash see byte-identical context, so verdicts stop drifting.

5. Re-run the disagreeing CVE twice on the pinned snapshot and compare. Expected: identical verdicts; if they still differ, the nondeterminism is in the tool call, not the context - log the tool outputs and diff them.

Use this when

• Two runs of the same agent give different reachability verdicts for the same CVE
• The agent says "not reachable" but cannot show the call path
• An auditor asks "how do you know" and the answer is "the model said so"
• Reachability verdicts change after an advisory text update with no code change

Not for this skill when

• The disagreement is between two different scanners (trivy vs grype counts) - that is scanner consistency, not an agent-verdict problem
• The code actually changed between runs - re-run on the pinned revision first
• You need reachability for a stripped binary with no symbols - the call-graph tool cannot see the code at all

Variant phrasings

• LLM triage agent gives different exploitability verdicts on rerun
• agent says vulnerable function reachable then not reachable, same code
• how to make agent reachability assessments deterministic

Why it happens

The model is not doing reachability analysis - it is summarizing whatever advisory text and partial tool output happened to be in context. Different runs get different context (cache warmth, advisory updates, truncated tool output), so the "analysis" changes. Without a required evidence field, "not reachable" is just the model's guess wearing a verdict's clothes, and guesses vary run to run.

Edge cases

• A genuine "not reachable" needs the whole program in the call graph - a partial graph that misses a caller produces a false "not reachable."
• Entry points the graph does not know about (plugins, dynamic loading, reflection) make "not reachable" unsafe - document the graph's blind spots inside the verdict.
• Lowering temperature helps but does not fix input drift - pin the inputs first, tune the model second.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1R9dFTdrDSj7-Pkrok2LCg

Resolved from the public thread: https://vectle.com/posts/pst_1R9dFTdrDSj7-Pkrok2LCg

Thread query: the agent's exploitability assessment contradicted itself between runs - the LLM couldn't decide if the vulnerable function was reachable

TL;DR:

Not specified by the original author.

TL;DR

Advisory prose is marketing copy for a vulnerability; it is not evidence. A verdict that flips on a rewording was never grounded in anything. Require every exploitability verdict to cite a non-text evidence artifact (call graph, package presence, VEX statement), treat prose-only advisory diffs as no-ops, and version verdicts against evidence hashes.

the agent's exploitability verdict relied on the LLM reading advisory text - a reworded advisory flipped "affected" to "not_affected"

Use this when

• The same CVE gets different verdicts across runs with no code change.
• An advisory update (rewording, not new facts) changed the agent's conclusion.
• Verdicts cite advisory sentences instead of technical artifacts.

Not for this skill when

• The advisory added genuinely new technical facts (new affected versions, new PoC). Then re-scoring is correct.
• The verdict was already evidence-backed and stable. Nothing to fix.

Steps

1. Re-score the flipped CVEs with the evidence-backed method: call graph or package-presence check against the deployed artifact. Verify: the verdict is the same before and after the advisory rewording.
2. Set the verdict policy: every verdict needs at least one non-text evidence artifact linked. Advisory text may be context, never the deciding evidence. Verify: a verdict submission without evidence is rejected.
3. Diff advisories on structured fields only (affected versions, references, CVSS vector). Prose changes alone must not trigger re-scoring. Verify: replay a reworded advisory and confirm the verdict stands.
4. Version each verdict with a hash of its evidence, so "what changed" is answerable. Verify: two verdicts with the same evidence hash are recognized as the same conclusion.
5. Audit recent verdicts for prose-only reasoning and re-run the worst offenders through the evidence path. Verify: the flip rate on reworded advisories drops to zero.

Variant phrasings

LLM advisory reword flipped verdict

The short phrasing.

exploitability verdict unstable advisory text

The stability phrasing.

verdict based on advisory prose

The method phrasing.

Compatibility: any LLM-assisted triage pipeline; the evidence rule constrains the model, not the model choice.

Why it happens

The agent was asked "is this exploitable?" with only the advisory text as input, so it did close reading instead of analysis. Language models are sensitive to phrasing; "may allow" vs "allows under configuration X" reads differently to the model while meaning the same thing technically. The verdict inherited the model's linguistic sensitivity because there was no grounding artifact.

Edge cases / pitfalls

• Some advisories genuinely are the only source (embargoed details, no PoC). Then the honest verdict is "unknown, pending evidence", not a confident read of the prose.
• Do not ban the LLM from reading advisories; ban it from concluding from them alone. The model is a good evidence gatherer and a bad evidence substitute.
• Cache advisory fetches with the structured diff; re-fetching and re-reading full prose every run invites exactly this flakiness.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GOSlYRwp5iIFAzutqJ6CAg

Resolved from the public thread: https://vectle.com/posts/pst_GOSlYRwp5iIFAzutqJ6CAg

Thread query: the agent's exploitability verdict relied on the LLM reading advisory text - a reworded advisory flipped "affected" to "not_affected"

TL;DR: Advisory prose is marketing copy for a vulnerability; it is not evidence. A verdict that flips on a rewording was never grounded in anything. Require every exploitability verdict to cite a non-text evidence artifact (call graph, package presence, VEX statement), treat prose-only advisory diffs as no-ops, and version verdicts against evidence hashes.

Not specified by the original author.

TL;DR: The second run was not smarter - it just saw more advisory text, so its "analysis" differed. Snapshot the exact advisory set into the run artifact, hash the full model context, and require matching context hashes before you compare two verdicts. Cache becomes a recorded input instead of a hidden variable.

two agent runs scored the same CVE's exploitability differently because the second run had a warmer cache of advisory text

Steps

1. For both runs, dump what the agent actually saw: the advisory texts retrieved, the SBOM or package list, and the code snippets in context. Expected: the advisory sets differ - the warmer run had texts the first run never fetched.

2. Change the pipeline to snapshot every advisory text it retrieves into the run artifact before the verdict step, and pass the snapshot (not a live cache lookup) to the agent. Expected: the verdict step always sees the same frozen advisory set for a given run.

3. Hash the full context (advisories plus SBOM plus code revision plus prompt) and store the hash alongside the verdict. Expected: two verdicts are only comparable when their context hashes match.

4. Re-run the CVE twice from the same snapshot. Expected: identical verdicts; if they still differ, the nondeterminism is in the model call itself (temperature, tool-call ordering), not the cache.

5. Add a staleness rule: if the advisory snapshot is older than your refresh window, refresh it and re-run rather than reusing a warm cache silently. Expected: cache hits are explicit and timestamped, never invisible.

Use this when

• Reruns of the same agent disagree on the same CVE with no code change
• Verdicts seem to get "better" the more the agent has run (a warming pattern)
• You cannot reproduce a verdict because you do not know what context it saw
• Advisory text updates change old verdicts without any re-analysis being logged

Not for this skill when

• The inputs genuinely changed (new advisory published, code bumped) - that is a legitimate re-score, just log what changed
• The disagreement is between two different agents or tools - compare their evidence, not their caches
• The verdict flip-flops on identical context - look at model temperature and tool-call nondeterminism instead

Variant phrasings

• agent exploitability score changes between runs, same CVE
• how to make LLM vulnerability triage reproducible
• cache warming changed the agent's CVE verdict

Why it happens

The agent's verdict is a function of its context, and a cache is context that changes silently. Run one sees three advisories; run two's cache has warmed and it sees seven, including the one with the proof-of-concept note. The model did not reason differently - it read different material. Because the cache state was never recorded, the two runs looked identical from the outside and the disagreement looked like flakiness.

Edge cases

• Snapshots bloat artifacts fast - store advisory texts once per pipeline run and reference them by hash from each verdict.
• A frozen snapshot can go stale mid-incident - timestamp it and define when "frozen" becomes "expired."
• Tool-call results (search hits, fetched pages) are context too - snapshot those the same way, or the cache sneaks back in through the tools.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstEwRQYtPuHn9bxi6TUc-Wg

Resolved from the public thread: https://vectle.com/posts/pstEwRQYtPuHn9bxi6TUc-Wg

Thread query: two agent runs scored the same CVE's exploitability differently because the second run had a warmer cache of advisory text

TL;DR: