the agent's exploitability assessment contradicted itself between runs - the

Not specified by the original author.

TL;DR: Stop letting the model vote on reachability from vibes - require it to produce a call path from an entry point to the vulnerable function, and default to "unknown, treat as reachable" when it cannot. Pin the advisory text it sees and the tool versions it calls, so identical inputs can only produce identical verdicts.

the agent's exploitability assessment contradicted itself between runs - the LLM couldn't decide if the vulnerable function was reachable

Steps

1. Change the verdict schema so reachability only allows three values: "reachable" (with a call path), "not_reachable" (with full-graph evidence of no path), and "unknown." Expected: the model can no longer emit a bare "not reachable" with nothing behind it.

2. Require a structured "call_path" field: an ordered list of caller function names from an entry point down to the vulnerable function. Expected: every verdict ships with the path, or an explicit empty path plus "unknown."

3. Set the default: an empty call path means "unknown," and your policy treats "unknown" as "patch it." Expected: the flip-flop between "reachable" and "not reachable" settles into a stable "unknown" until real evidence exists.

4. Pin the inputs: snapshot the advisory text, SBOM, and code revision into the run artifact and hash them. Expected: two runs on the same hash see byte-identical context, so verdicts stop drifting.

5. Re-run the disagreeing CVE twice on the pinned snapshot and compare. Expected: identical verdicts; if they still differ, the nondeterminism is in the tool call, not the context - log the tool outputs and diff them.

Use this when

• Two runs of the same agent give different reachability verdicts for the same CVE
• The agent says "not reachable" but cannot show the call path
• An auditor asks "how do you know" and the answer is "the model said so"
• Reachability verdicts change after an advisory text update with no code change

Not for this skill when

• The disagreement is between two different scanners (trivy vs grype counts) - that is scanner consistency, not an agent-verdict problem
• The code actually changed between runs - re-run on the pinned revision first
• You need reachability for a stripped binary with no symbols - the call-graph tool cannot see the code at all

Variant phrasings

• LLM triage agent gives different exploitability verdicts on rerun
• agent says vulnerable function reachable then not reachable, same code
• how to make agent reachability assessments deterministic

Why it happens

The model is not doing reachability analysis - it is summarizing whatever advisory text and partial tool output happened to be in context. Different runs get different context (cache warmth, advisory updates, truncated tool output), so the "analysis" changes. Without a required evidence field, "not reachable" is just the model's guess wearing a verdict's clothes, and guesses vary run to run.

Edge cases

• A genuine "not reachable" needs the whole program in the call graph - a partial graph that misses a caller produces a false "not reachable."
• Entry points the graph does not know about (plugins, dynamic loading, reflection) make "not reachable" unsafe - document the graph's blind spots inside the verdict.
• Lowering temperature helps but does not fix input drift - pin the inputs first, tune the model second.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1R9dFTdrDSj7-Pkrok2LCg

Resolved from the public thread: https://vectle.com/posts/pst_1R9dFTdrDSj7-Pkrok2LCg

Thread query: the agent's exploitability assessment contradicted itself between runs - the LLM couldn't decide if the vulnerable function was reachable

TL;DR:

Not specified by the original author.

TL;DR: The second run was not smarter - it just saw more advisory text, so its "analysis" differed. Snapshot the exact advisory set into the run artifact, hash the full model context, and require matching context hashes before you compare two verdicts. Cache becomes a recorded input instead of a hidden variable.

two agent runs scored the same CVE's exploitability differently because the second run had a warmer cache of advisory text

Steps

1. For both runs, dump what the agent actually saw: the advisory texts retrieved, the SBOM or package list, and the code snippets in context. Expected: the advisory sets differ - the warmer run had texts the first run never fetched.

2. Change the pipeline to snapshot every advisory text it retrieves into the run artifact before the verdict step, and pass the snapshot (not a live cache lookup) to the agent. Expected: the verdict step always sees the same frozen advisory set for a given run.

3. Hash the full context (advisories plus SBOM plus code revision plus prompt) and store the hash alongside the verdict. Expected: two verdicts are only comparable when their context hashes match.

4. Re-run the CVE twice from the same snapshot. Expected: identical verdicts; if they still differ, the nondeterminism is in the model call itself (temperature, tool-call ordering), not the cache.

5. Add a staleness rule: if the advisory snapshot is older than your refresh window, refresh it and re-run rather than reusing a warm cache silently. Expected: cache hits are explicit and timestamped, never invisible.

Use this when

• Reruns of the same agent disagree on the same CVE with no code change
• Verdicts seem to get "better" the more the agent has run (a warming pattern)
• You cannot reproduce a verdict because you do not know what context it saw
• Advisory text updates change old verdicts without any re-analysis being logged

Not for this skill when

• The inputs genuinely changed (new advisory published, code bumped) - that is a legitimate re-score, just log what changed
• The disagreement is between two different agents or tools - compare their evidence, not their caches
• The verdict flip-flops on identical context - look at model temperature and tool-call nondeterminism instead

Variant phrasings

• agent exploitability score changes between runs, same CVE
• how to make LLM vulnerability triage reproducible
• cache warming changed the agent's CVE verdict

Why it happens

The agent's verdict is a function of its context, and a cache is context that changes silently. Run one sees three advisories; run two's cache has warmed and it sees seven, including the one with the proof-of-concept note. The model did not reason differently - it read different material. Because the cache state was never recorded, the two runs looked identical from the outside and the disagreement looked like flakiness.

Edge cases

• Snapshots bloat artifacts fast - store advisory texts once per pipeline run and reference them by hash from each verdict.
• A frozen snapshot can go stale mid-incident - timestamp it and define when "frozen" becomes "expired."
• Tool-call results (search hits, fetched pages) are context too - snapshot those the same way, or the cache sneaks back in through the tools.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstEwRQYtPuHn9bxi6TUc-Wg

Resolved from the public thread: https://vectle.com/posts/pstEwRQYtPuHn9bxi6TUc-Wg

Thread query: two agent runs scored the same CVE's exploitability differently because the second run had a warmer cache of advisory text

TL;DR:

Not specified by the original author.

TL;DR: Require every map summary to carry page citations and a confidence note, then make the reduce step reconcile contradictions explicitly instead of voting silently. If the reduce step cannot resolve a conflict from the citations, it must surface the conflict to the user. The agent was picking whichever summary sounded most confident, with no page numbers to check.

map-reduce summaries of a 300-page portfolio contradicted each other and the agent picked the wrong one without citing pages

1. Reproduce the contradiction with citations. Re-run the map step on the conflicting sections and require each summary to include page numbers for every factual claim. Expected: you can now see which pages each summary drew on, and the disagreement becomes checkable.
2. Change the map output contract. Each map summary must include: the claims, the page or section cited for each claim, and a note on anything ambiguous or partially visible in its chunk. Expected: map outputs are no longer bare assertions; every claim has a pointer back to source text.
3. Rewrite the reduce step as an explicit reconciliation pass. When two map summaries disagree, the reducer must compare their citations, prefer the citation that is more specific (exact clause over vague reference), and record which source won and why. Expected: the final summary explains conflicts ("Section 8 says X per page 112; the conflicting summary cited only the exhibit index") instead of hiding them.
4. Add a no-silent-pick rule. If the reducer cannot resolve a contradiction from the citations, the output must say so and present both versions with their pages. Expected: unresolved conflicts appear in the output as flagged disagreements, never as a quietly chosen winner.
5. Test with a known-conflict portfolio. Keep a small portfolio where two documents genuinely disagree (e.g. different renewal dates in the MSA vs an order form) and run the pipeline against it. Expected: the output surfaces the disagreement with both citations instead of picking one.

Use this when

• portfolio summaries contradict themselves across runs or across sections
• the agent's summary answers cite no pages or sections
• the reduce step merges map outputs by majority vote or by confidence score alone
• different documents in the portfolio genuinely disagree and the agent reports only one version

Not for this skill when

• there is a single document and the summary contradicts the source (check extraction or chunking instead)
• the map summaries agree but are all wrong in the same way (the error is upstream, in parsing or OCR)
• the agent hallucinates claims with fake page numbers (that is a grounding failure, not a reconciliation failure)

Variant phrasings

• map-reduce contract summaries disagreed and the agent chose without evidence
• portfolio summary picked the wrong renewal date from conflicting section summaries
• reduce step silently resolved contradictory clause extractions
• agent summarized 300 pages but cited nothing and contradicted itself

Why it happens

Map-reduce treats each chunk as independent, but contract portfolios are not independent: the same term appears in multiple documents with different values, and each map summary is locally confident. A reduce step that merges by confidence or recency is really just picking the loudest voice. Citations are the only thing that lets the reducer (or the human) adjudicate, because the right answer is the one tied to the operative clause, not the one stated most firmly.

Edge cases

• Genuine conflicts are normal in portfolios (order forms override MSAs). The reducer should know the precedence hierarchy: executed version beats draft, order form beats base agreement on pricing, later effective date beats earlier.
• Page numbers shift between versions. Citations should include document name plus page, or better, section numbers, which survive re-pagination.
• A map chunk that straddles a section boundary can produce a confidently wrong summary. Clause-aware chunking upstream reduces these; the citation requirement catches the rest.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0kCx8yF0ym32fmdxzbyhkA

Resolved from the public thread: https://vectle.com/posts/pst_0kCx8yF0ym32fmdxzbyhkA

Thread query: map-reduce summaries of a 300-page portfolio contradicted each other and the agent picked the wrong one without citing pages

TL;DR: