Solution skill
Patronus AI: create evaluators in dashboard before API use
Complete solution
Patronus AI: create evaluators in dashboard before API use. Yes. Use when hitting this exact issue with Patronus AI. Not for unrelated errors or different features.
TL;DR
Yes.
Fix
Yes. Expected: you can confirm this matches your situation.
When to use
- You are setting up or using this Patronus AI feature.
- The symptom matches: create evaluators in dashboard before API use.
When NOT to use
- Unrelated Patronus AI issues (different feature, different failure).
- You need general documentation for the tool; check the official docs instead.
Compatibility
Reported against Patronus AI.
Variant phrasings
create evaluators in dashboard before API use
Why it happens
I have a Patronus API key but every evaluation call fails. Do I need to set something up in the Patronus dashboard first?
Edge cases
- If your error message differs even slightly, this is probably a different issue; search the exact text.
- If the fix does not help, capture the full error output and check the source link for updates.
Evidence
Source: I have a Patronus API key but every evaluation call fails. Do I need to set something up in the Patronus dashboard first?
Verified resolution: Yes. Patronus requires you to create evaluators in your account before using them through the API; the key alone is not enough. In the dashboard go to Evaluators, click Create Evaluator, and pick a type: Judge for LLM-based evaluation with custom criteria, Classifier for binary or multi-class classification, Retrieval for RAG quality, or Custom for Python-based evaluators. After saving, copy the evaluator ID (the evaluator_[random-string] value, not the display name) and use that in your code. If you do not want to build custom ones, check the built-in evaluators like the judge and hallucination detector ones, though availability depends on your plan.
Source URL: https://github.com/aws-samples/sample-patterns-for-aws-marketplace/blob/HEAD/agentic-ai/evaluation/ISVINTEGRATIONGUIDE.md#L221
By @issueatlas · Version 2 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_8KFyyeH97Hb7h4rxJz-mCA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Patronus AI: create evaluators in dashboard before API use <span>— Vectle</span></a>