Immutable solution version
Patronus AI: create evaluators in dashboard before API use
Complete solution
Yes.
Yes. Patronus requires you to create evaluators in your account before using them through the API; the key alone is not enough. In the dashboard go to Evaluators, click Create Evaluator, and pick a type: Judge for LLM-based evaluation with custom criteria, Classifier for binary or multi-class classification, Retrieval for RAG quality, or Custom for Python-based evaluators. After saving, copy the evaluator ID (the evaluator_[random-string] value, not the display name) and use that in your code. If you do not want to build custom ones, check the built-in evaluators like the judge and hallucination detector ones, though availability depends on your plan.
Context: I have a Patronus API key but every evaluation call fails. Do I need to set something up in the Patronus dashboard first?
Evidence
Source: I have a Patronus API key but every evaluation call fails. Do I need to set something up in the Patronus dashboard first?
Verified resolution: Yes. Patronus requires you to create evaluators in your account before using them through the API; the key alone is not enough. In the dashboard go to Evaluators, click Create Evaluator, and pick a type: Judge for LLM-based evaluation with custom criteria, Classifier for binary or multi-class classification, Retrieval for RAG quality, or Custom for Python-based evaluators. After saving, copy the evaluator ID (the evaluator_[random-string] value, not the display name) and use that in your code. If you do not want to build custom ones, check the built-in evaluators like the judge and hallucination detector ones, though availability depends on your plan.
Source URL: https://github.com/aws-samples/sample-patterns-for-aws-marketplace/blob/HEAD/agentic-ai/evaluation/ISVINTEGRATIONGUIDE.md#L221
By @issueatlas · Version 1 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_8KFyyeH97Hb7h4rxJz-mCA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Patronus AI: create evaluators in dashboard before API use <span>— Vectle</span></a>