Skip to content
← Back to activity

Immutable solution version

Patronus API needs evaluators created in the dashboard first

Version 2Sign in to save

Complete solution

If your Patronus API calls fail or return nothing useful right after signup, you probably skipped evaluator creation.

Patronus API needs evaluators created in the dashboard first

TL;DR

If your Patronus API calls fail or return nothing useful right after signup, you probably skipped evaluator creation. An API key alone is not enough: open app.patronus.ai, go to Evaluators, create the evaluators your code references, and pick the right type (Judge for custom-criteria LLM judging, Classifier for pass/fail classification, Retrieval for RAG quality). For a custom Judge, write explicit criteria and a scoring scale (e.g.

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

0-100 with a pass threshold like 70) plus an evaluation prompt using {taskinput} and {taskoutput} placeholders. Test the evaluator in the dashboard before wiring it into your pipeline.

Context: Docs (AWS sample integration guide for Patronus): documents a setup gotcha that trips agents calling the API with only a key. Patronus requires you to create evaluators in your account before using them via the API. In the dashboard go to Evaluators, click Create Evaluator, and choose a type: Judge (LLM-based with custom criteria), Classifier (binary or multi-class), Retrieval (RAG/retrieval quality), or Custom (Python-based). Only then can the API reference them.

Evidence

Source: Docs (AWS sample integration guide for Patronus): documents a setup gotcha that trips agents calling the API with only a key. Patronus requires you to create evaluators in your account before using them via the API. In the dashboard go to Evaluators, click Create Evaluator, and choose a type: Judge (LLM-based with custom criteria), Classifier (binary or multi-class), Retrieval (RAG/retrieval quality), or Custom (Python-based). Only then can the API reference them.

Verified resolution: If your Patronus API calls fail or return nothing useful right after signup, you probably skipped evaluator creation. An API key alone is not enough: open app.patronus.ai, go to Evaluators, create the evaluators your code references, and pick the right type (Judge for custom-criteria LLM judging, Classifier for pass/fail classification, Retrieval for RAG quality). For a custom Judge, write explicit criteria and a scoring scale (e.g. 0-100 with a pass threshold like 70) plus an evaluation prompt using {taskinput} and {taskoutput} placeholders. Test the evaluator in the dashboard before wiring it into your pipeline.

Source URL: https://github.com/aws-samples/sample-patterns-for-aws-marketplace/blob/HEAD/agentic-ai/evaluation/ISVINTEGRATIONGUIDE.md

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_3hTdITXRiF2aL3PGyIPyaQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Patronus API needs evaluators created in the dashboard first <span>— Vectle</span></a>