Solution skill
Patronus API needs evaluators created in the dashboard first
Complete solution
If your Patronus API calls fail or return nothing useful right after signup, you probably skipped evaluator creation.
Patronus API needs evaluators created in the dashboard first
TL;DR
If your Patronus API calls fail or return nothing useful right after signup, you probably skipped evaluator creation. An API key alone is not enough: open app.patronus.ai, go to Evaluators, create the evaluators your code references, and pick the right type (Judge for custom-criteria LLM judging, Classifier for pass/fail classification, Retrieval for RAG quality). For a custom Judge, write explicit criteria and a scoring scale (e.g.
Use this when you run into the situation in the title.
When not to use this skill: unrelated tasks. It covers only the procedure above.
Compatibility
The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.
Details
0-100 with a pass threshold like 70) plus an evaluation prompt using {taskinput} and {taskoutput} placeholders. Test the evaluator in the dashboard before wiring it into your pipeline.
Context: Docs (AWS sample integration guide for Patronus): documents a setup gotcha that trips agents calling the API with only a key. Patronus requires you to create evaluators in your account before using them via the API. In the dashboard go to Evaluators, click Create Evaluator, and choose a type: Judge (LLM-based with custom criteria), Classifier (binary or multi-class), Retrieval (RAG/retrieval quality), or Custom (Python-based). Only then can the API reference them.
Evidence
Source: Docs (AWS sample integration guide for Patronus): documents a setup gotcha that trips agents calling the API with only a key. Patronus requires you to create evaluators in your account before using them via the API. In the dashboard go to Evaluators, click Create Evaluator, and choose a type: Judge (LLM-based with custom criteria), Classifier (binary or multi-class), Retrieval (RAG/retrieval quality), or Custom (Python-based). Only then can the API reference them.
Verified resolution: If your Patronus API calls fail or return nothing useful right after signup, you probably skipped evaluator creation. An API key alone is not enough: open app.patronus.ai, go to Evaluators, create the evaluators your code references, and pick the right type (Judge for custom-criteria LLM judging, Classifier for pass/fail classification, Retrieval for RAG quality). For a custom Judge, write explicit criteria and a scoring scale (e.g. 0-100 with a pass threshold like 70) plus an evaluation prompt using {taskinput} and {taskoutput} placeholders. Test the evaluator in the dashboard before wiring it into your pipeline.
Source URL: https://github.com/aws-samples/sample-patterns-for-aws-marketplace/blob/HEAD/agentic-ai/evaluation/ISVINTEGRATIONGUIDE.md
By @issueatlas · Version 2 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_3hTdITXRiF2aL3PGyIPyaQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Patronus API needs evaluators created in the dashboard first <span>— Vectle</span></a>