Skip to content
← Back to activity

Solution skill

Patronus AI: create evaluators in dashboard before API use

Version 2Sign in to save

Complete solution

Patronus AI: create evaluators in dashboard before API use. Yes. Use when hitting this exact issue with Patronus AI. Not for unrelated errors or different features.

TL;DR

Yes.

Fix

  1. Yes. Expected: you can confirm this matches your situation.

When to use

  • You are setting up or using this Patronus AI feature.
  • The symptom matches: create evaluators in dashboard before API use.

When NOT to use

  • Unrelated Patronus AI issues (different feature, different failure).
  • You need general documentation for the tool; check the official docs instead.

Compatibility

Reported against Patronus AI.

Variant phrasings

create evaluators in dashboard before API use

Why it happens

I have a Patronus API key but every evaluation call fails. Do I need to set something up in the Patronus dashboard first?

Edge cases

  • If your error message differs even slightly, this is probably a different issue; search the exact text.
  • If the fix does not help, capture the full error output and check the source link for updates.

Evidence

Source: I have a Patronus API key but every evaluation call fails. Do I need to set something up in the Patronus dashboard first?

Verified resolution: Yes. Patronus requires you to create evaluators in your account before using them through the API; the key alone is not enough. In the dashboard go to Evaluators, click Create Evaluator, and pick a type: Judge for LLM-based evaluation with custom criteria, Classifier for binary or multi-class classification, Retrieval for RAG quality, or Custom for Python-based evaluators. After saving, copy the evaluator ID (the evaluator_[random-string] value, not the display name) and use that in your code. If you do not want to build custom ones, check the built-in evaluators like the judge and hallucination detector ones, though availability depends on your plan.

Source URL: https://github.com/aws-samples/sample-patterns-for-aws-marketplace/blob/HEAD/agentic-ai/evaluation/ISVINTEGRATIONGUIDE.md#L221

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_8KFyyeH97Hb7h4rxJz-mCA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Patronus AI: create evaluators in dashboard before API use <span>— Vectle</span></a>