Stream OpenAI responses robustly: assemble events, handle cutoffs, never trust partial state

Streaming without the footguns

The steps

1. Stream for user experience, not for speed. Streaming does not change total generation time; it reduces time to first token. Use it where partial progress matters to the user.
2. Assemble the full message from events before acting on it. Buffer text deltas, tool call deltas, and annotations; only when the stream signals completion do you have a message. Executing a tool call from a partial delta is a classic bug.
3. Handle the cutoff: if the connection drops mid-stream, you have a partial response and no completion signal. Decide the policy up front: resume, retry the whole request, or surface the partial with a clear "incomplete" marker. Never silently present a truncated answer as complete.
4. Set timeouts at two levels: time to first token (catches a hung request early) and total stream time (catches a stream that trickles forever). Both need values before production.
5. Do not stream structured outputs into parsers incrementally. Parse once, after assembly, with the same validation and refusal checks as a non-streamed call.
6. Log stream anomalies: unexpected event order, missing completion signals, mid-stream errors. These are the diagnostics that explain the one-in-a-thousand weird response.

The trap

Treating the stream as the message. Until the completion event arrives, everything is provisional. Code that acts on deltas as they arrive will eventually act on half a thought.

Checklist

• Full assembly before any action on the content.
• Dropped-connection policy defined and tested.
• First-token and total-stream timeouts set.
• Structured outputs parsed after assembly, validated as usual.
• Stream anomalies logged for diagnosis.

OpenAI's production best practices guide documents that stream true reduces time to first token without changing total time. Verified live at https://developers.openai.com/api/docs/guides/production-best-practices.