OpenAI Batch API at scale: build the request file, handle the 24h window, harvest partial results

Batch, done right

The steps

1. Build the request file as JSONL: one line per request with a custom_id you generate. The custom id is how you join results back to your records, so make it your own primary key, not a sequence number you will lose.
2. Validate every line before upload: each is a complete API request object, each custom_id is unique. One malformed line can poison the whole file.
3. Upload the file, then create the batch with the right endpoint and completion window. Note the batch id immediately; that id is your handle for everything after.
4. Poll for status, but poll politely: statuses move from validating to in_progress to finalizing to completed. Back off between polls; hammering the status endpoint helps nothing.
5. When it completes, download the output file via output_file_id and join results on custom_id. Do this join defensively: some requests may have errored individually even when the batch completed.
6. Check the error_file_id too. Individual request failures land there with their custom ids. Retry just those, not the whole batch.
7. Handle expired: a batch that cannot finish inside the 24-hour window expires. Design for this: split huge jobs into smaller batches so one slow shard does not sink the rest.

The trap

Putting latency-sensitive work in batch, or assuming completed means every request succeeded. Batch is for throughput, not speed, and partial failure is normal at scale.

Checklist

• custom_id values are your own join keys, unique per line.
• The request file is validated before upload.
• Both outputfileid and errorfileid are consumed.
• Expired batches are split and resubmitted in smaller pieces.
• Nothing user-facing waits on a batch.

OpenAI's Batch guide documents 50% lower costs, the 24-hour turnaround, the validating/inprogress/finalizing/completed/expired statuses, and outputfileid/errorfile_id. Verified live at https://developers.openai.com/api/docs/guides/batch.