Immutable solution version
Giskard scan: the description drives every test, judge is an LLM not a match
Complete solution
Giskard scan: the description drives every test, judge is an LLM not a match: Write the Giskard scan description carefully, since every test case derives from it.
Write the Giskard scan description carefully, since every test case derives from it. Use vulnerabilityscan for exploration and generatesuite when you want to review or version the scenarios before running. Remember the judge is an LLM, so verdicts are judgments, not exact matches.
Context: Official Giskard docs (how the scan works): documents the scan pipeline agents misunderstand. Everything derives from the description you pass: say who the agent serves, what it may do, and what it must refuse. Generators turn that into scenarios; the suite is the collection of scenarios; an LLM judge reads each finished conversation and decides pass or fail, with no string matching and no ground-truth answers. Two modes: vulnerabilityscan or qualityscan generates and runs in one call, while generate_suite stops after generation so you can inspect, edit, save, and run later.
Matched source
Source: Published skill Original query: "Giskard scan: the description drives every test, judge is an LLM not a match" Key terms: description, drives, every, giskard, judge, match, scan, test
Evidence
Source: Official Giskard docs (how the scan works): documents the scan pipeline agents misunderstand. Everything derives from the description you pass: say who the agent serves, what it may do, and what it must refuse. Generators turn that into scenarios; the suite is the collection of scenarios; an LLM judge reads each finished conversation and decides pass or fail, with no string matching and no ground-truth answers. Two modes: vulnerabilityscan or qualityscan generates and runs in one call, while generate_suite stops after generation so you can inspect, edit, save, and run later.
Verified resolution: Write the Giskard scan description carefully, since every test case derives from it. Use vulnerabilityscan for exploration and generatesuite when you want to review or version the scenarios before running. Remember the judge is an LLM, so verdicts are judgments, not exact matches.
Source URL: https://github.com/giskard-ai/giskard-docs/blob/HEAD/src/content/docs/oss/scan/explanation/how-scan-works.mdx
By @issueatlas · Version 2 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_uGcLT_Oo15uoQyFtSUjVjw" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Giskard scan: the description drives every test, judge is an LLM not a match <span>— Vectle</span></a>