FastAPI: relay a browser client to the Realtime API over websockets

TL;DR

Use a relay when your server captures audio or must keep the project API key server-side while the browser only holds an ephemeral secret. Verified against the Realtime websocket docs on 2026-09-26. Wait for session.started before sending audio or commands. It carries the resolved session config and session id.

When to use

You are seeing this: Verified against the Realtime websocket docs on 2026-09-26. Use this skill when you run into "FastAPI: relay a browser client to the Realtime API over websockets".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

FastAPI Realtime websocket relay

Use a relay when your server captures audio or must keep the project API key server-side while the browser only holds an ephemeral secret. Verified against the Realtime websocket docs on 2026-09-26.

Connection setup

1. Open a server-side websocket to the Realtime endpoint with the header Authorization: Bearer plus your server-side key. (Keep the key out of any client-visible code; the relay socket lives entirely on the server.)

2. Send session.start as the FIRST message, with model, instructions, audio format, and voice inside the session object.

3. Wait for session.started before sending audio or commands. It carries the resolved session config and session id. Sending audio first is silently dropped.

Audio framing

• Websocket audio is manual: send JSON events with base64-encoded PCM16 mono audio at 24 kHz via input_audio_buffer.append.

• Read response.audio.delta events from OpenAI and forward the base64 chunks to the browser client over your own websocket.

• In FastAPI, use the websockets library (or any ASGI websocket) for the upstream socket, and pair it with the client websocket in an async relay loop.

Traps

• Base64 your audio; raw bytes in a JSON event corrupt the frame.

• Handle backpressure: if the browser is slow, pause reading upstream rather than buffering unbounded audio in memory.

• On either socket closing, close the other one. A half-open relay holds the OpenAI session (and its cost) open.

Check before you ship

• Log the first five upstream messages; the sequence must be session.start, then session.started, then audio.

• Test a mid-call disconnect and confirm the upstream socket closes within seconds.

OpenAI's Realtime websocket guide documents connecting with an Authorization Bearer header, sending session.start first with config in the session object, waiting for session.started, and manually managing the input audio buffer with base64-encoded audio for websocket transport.