Stream OpenAI responses robustly: assemble events, handle cutoffs, never trust partial state

TL;DR

Use it where partial progress matters to the user. Stream for user experience, not for speed. Streaming does not change total generation time; it reduces time to first token. Set timeouts at two levels: time to first token (catches a hung request early) and total stream time (catches a stream that trickles forever).

When to use

You are seeing this: Stream for user experience, not for speed. Use this skill when you run into "Stream OpenAI responses robustly: assemble events, handle cutoffs, never trust partial state".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Streaming without the footguns

The steps

1. Stream for user experience, not for speed. Streaming does not change total generation time; it reduces time to first token. Use it where partial progress matters to the user.
2. Assemble the full message from events before acting on it. Buffer text deltas, tool call deltas, and annotations; only when the stream signals completion do you have a message. Executing a tool call from a partial delta is a classic bug.
3. Handle the cutoff: if the connection drops mid-stream, you have a partial response and no completion signal. Decide the policy up front: resume, retry the whole request, or surface the partial with a clear "incomplete" marker. Never silently present a truncated answer as complete.
4. Set timeouts at two levels: time to first token (catches a hung request early) and total stream time (catches a stream that trickles forever). Both need values before production.
5. Do not stream structured outputs into parsers incrementally. Parse once, after assembly, with the same validation and refusal checks as a non-streamed call.
6. Log stream anomalies: unexpected event order, missing completion signals, mid-stream errors. These are the diagnostics that explain the one-in-a-thousand weird response.

The trap

Treating the stream as the message. Until the completion event arrives, everything is provisional. Code that acts on deltas as they arrive will eventually act on half a thought.

Checklist

• Full assembly before any action on the content.
• Dropped-connection policy defined and tested.
• First-token and total-stream timeouts set.
• Structured outputs parsed after assembly, validated as usual.
• Stream anomalies logged for diagnosis.

OpenAI's production best practices guide documents that stream true reduces time to first token without changing total time. Verified live at https://developers.openai.com/api/docs/guides/production-best-practices.