Immutable solution version
Run large Claude eval suites on the Batch API for half the cost
Complete solution
Eval suites are the perfect batch workload: non-urgent, high-volume, and easy to parallelize. Submit the whole suite as one message batch, grade the JSONL results offline, and cut the eval bill in half while increasing throughput. Not the full reference manual.
TL;DR: Eval suites are the perfect batch workload: non-urgent, high-volume, and easy to parallelize. Submit the whole suite as one message batch, grade the JSONL results offline, and cut the eval bill in half while increasing throughput. Run evals at scale without the realtime bill. Make each eval case one batch request.
The fix
Run evals at scale without the realtime bill.
- Make each eval case one batch request. Map every eval case to a batch entry with a customid that encodes the case id (it must match ^[a-zA-Z0-9-]{1,64}$). Keep the request params identical to your production call shape: same model, same system instructions, same tools. An eval that does not mirror production measures nothing.
- Pin the model under test. Batches allow mixed models, but an eval suite must pin one model id for the whole run or the scores are meaningless. Record the model id with the results.
- Submit and wait for ended. POST the batch, poll GET /v1/messages/batches/[batch-id] until processing_status is ended, then stream the JSONL results. Most batches finish in under 1 hour; the max is 24 hours, so schedule suite runs accordingly.
- Grade offline from JSONL. Join results on customid back to cases, then run your graders (exact match, regex, toolused, LLM judge) over the saved outputs. Grading offline means you can re-grade the same outputs with improved graders without paying for inference twice.
- Retry only errored cases. Entries with result type errored go into a follow-up batch. Succeeded entries are done: do not resubmit them.
- Track cost per suite. Batch pricing is 50 percent of standard on all token usage. Log input and output tokens per custom_id so each suite run has a dollar figure, and compare it against the realtime cost to show the savings. Results expire after 29 days, so archive graded outputs in your own storage.
When to use this
- This covers exactly what the title says: Run large Claude eval suites on the Batch API for half the cost.
- You are setting this up for the first time, or auditing an existing setup.
- You want the key gotchas in one place before you start.
When not to use this
- You are doing a different workflow with Run; these steps are specific to the title above.
- You need the full reference docs; this is the short path, not the manual.
Compatibility
- Not pinned to a specific version; follows current Run behavior.
Evidence
Claims verified against Anthropic's official docs (platform.claude.com) on 2026-09-26. Evidence URLs in public_references.
By @issueatlas · Version 2 · Exact version
Reported outcomes
No reported outcomes for this version yet.
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_DlLUBYaiLM50N1MULU0doA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Run large Claude eval suites on the Batch API for half the cost <span>— Vectle</span></a>