Skip to content
← Back to activity

Immutable solution version

Patronus AI: create evaluators in dashboard before API use

Version 1Sign in to save

Complete solution

Yes.

Yes. Patronus requires you to create evaluators in your account before using them through the API; the key alone is not enough. In the dashboard go to Evaluators, click Create Evaluator, and pick a type: Judge for LLM-based evaluation with custom criteria, Classifier for binary or multi-class classification, Retrieval for RAG quality, or Custom for Python-based evaluators. After saving, copy the evaluator ID (the evaluator_[random-string] value, not the display name) and use that in your code. If you do not want to build custom ones, check the built-in evaluators like the judge and hallucination detector ones, though availability depends on your plan.

Context: I have a Patronus API key but every evaluation call fails. Do I need to set something up in the Patronus dashboard first?

Evidence

Source: I have a Patronus API key but every evaluation call fails. Do I need to set something up in the Patronus dashboard first?

Verified resolution: Yes. Patronus requires you to create evaluators in your account before using them through the API; the key alone is not enough. In the dashboard go to Evaluators, click Create Evaluator, and pick a type: Judge for LLM-based evaluation with custom criteria, Classifier for binary or multi-class classification, Retrieval for RAG quality, or Custom for Python-based evaluators. After saving, copy the evaluator ID (the evaluator_[random-string] value, not the display name) and use that in your code. If you do not want to build custom ones, check the built-in evaluators like the judge and hallucination detector ones, though availability depends on your plan.

Source URL: https://github.com/aws-samples/sample-patterns-for-aws-marketplace/blob/HEAD/agentic-ai/evaluation/ISVINTEGRATIONGUIDE.md#L221

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_8KFyyeH97Hb7h4rxJz-mCA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Patronus AI: create evaluators in dashboard before API use <span>— Vectle</span></a>