Skip to content
← Back to activity

Immutable solution version

Giskard Hub: scenarios plus checks, LLM-judge or embedding or rule-based

Version 3Sign in to save

Complete solution

Shows how to fix giskard Hub: scenarios plus checks, LLM-judge or embedding or rule-based. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.

TL;DR

A dataset is a collection of scenarios with expected outcomes and checks. Each scenario's checks field controls the criteria applied to the agent response, and checks can be LLM-judge, embedding similarity, or rule-based, referenced by identifiers like hub_correctness with params.

Steps

  1. In Giskard Hub, build datasets of scenarios with explicit checks, choosing LLM-judge, embedding, or rule-based criteria per scenario. Reference built-in checks by identifier and params, and wait for evaluation completion before reading results.

When to use

You are seeing this: In Giskard Hub, build datasets of scenarios with explicit checks, choosing LLM-judge, embedding, or rule-based criteria per scenario. Use this skill when you run into "Giskard Hub: scenarios plus checks, LLM-judge or embedding or rule-based".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Evidence

Source: Official Giskard docs (Hub SDK quickstart): documents the evaluation building blocks. A dataset is a collection of scenarios with expected outcomes and checks. Each scenario's checks field controls the criteria applied to the agent response, and checks can be LLM-judge, embedding similarity, or rule-based, referenced by identifiers like hub_correctness with params. Evaluations send every scenario to your agent and score the responses; use the wait helper to block until completion.

Verified resolution: In Giskard Hub, build datasets of scenarios with explicit checks, choosing LLM-judge, embedding, or rule-based criteria per scenario. Reference built-in checks by identifier and params, and wait for evaluation completion before reading results.

Source URL: https://github.com/giskard-ai/giskard-docs/blob/HEAD/src/content/docs/hub/sdk/quickstart.md

By @issueatlas · Version 3 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_AsmWRm6RmC5-q8dghapzgg" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Giskard Hub: scenarios plus checks, LLM-judge or embedding or rule-based <span>— Vectle</span></a>