Skip to content
← Back to activity

Immutable solution version

Run large Claude eval suites on the Batch API for half the cost

Version 1Sign in to save

Complete solution

Eval suites are the perfect batch workload: non-urgent, high-volume, and easy to parallelize. Submit the whole suite as one message batch, grade the JSONL results offline, and cut the eval bill in half while increasing throughput.

Run evals at scale without the realtime bill.

  1. Make each eval case one batch request. Map every eval case to a batch entry with a customid that encodes the case id (it must match ^[a-zA-Z0-9-]{1,64}$). Keep the request params identical to your production call shape: same model, same system instructions, same tools. An eval that does not mirror production measures nothing.
  2. Pin the model under test. Batches allow mixed models, but an eval suite must pin one model id for the whole run or the scores are meaningless. Record the model id with the results.
  3. Submit and wait for ended. POST the batch, poll GET /v1/messages/batches/[batch-id] until processing_status is ended, then stream the JSONL results. Most batches finish in under 1 hour; the max is 24 hours, so schedule suite runs accordingly.
  4. Grade offline from JSONL. Join results on customid back to cases, then run your graders (exact match, regex, toolused, LLM judge) over the saved outputs. Grading offline means you can re-grade the same outputs with improved graders without paying for inference twice.
  5. Retry only errored cases. Entries with result type errored go into a follow-up batch. Succeeded entries are done: do not resubmit them.
  6. Track cost per suite. Batch pricing is 50 percent of standard on all token usage. Log input and output tokens per custom_id so each suite run has a dollar figure, and compare it against the realtime cost to show the savings. Results expire after 29 days, so archive graded outputs in your own storage.

Evidence

Claims verified against Anthropic's official docs (platform.claude.com) on 2026-09-26. Evidence URLs in public_references.

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_DlLUBYaiLM50N1MULU0doA" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Run large Claude eval suites on the Batch API for half the cost <span>— Vectle</span></a>