Skip to content
← Back to activity

Immutable solution version

Wire an OpenAI Realtime voice agent: ephemeral keys, session setup, turn handling

Version 1Sign in to save

Complete solution

A realtime voice agent is a session lifecycle, not a request. The ordering matters: mint the ephemeral key server-side, configure the session, then handle audio turns, interruptions, and tool calls inside it.

Realtime voice, in order

The steps

  1. Mint an ephemeral client secret on your server for each session. The browser or device connects with that short-lived secret; your long-lived API key never leaves the server.
  2. Create the session from the client over WebRTC (browser) or WebSocket (server). Configure the session up front: voice, input audio transcription, and the tools the agent may call.
  3. Handle turns as events, not requests. The model emits audio deltas, transcriptions, and tool calls as a stream of session events; your code reacts to each event type and keeps local state in sync.
  4. Implement interruption: when the user starts speaking over the agent, stop playback and truncate the conversation at the interruption point so the model does not keep answering a stale turn.
  5. Wire tool calls through the session: when the model requests a function, run it in your backend and send the result back into the session as a conversation item. Voice latency budgets are tight, so keep tool execution fast or acknowledge the delay in speech.
  6. Decide the architecture deliberately. Use the Realtime API when the interaction must feel conversational with barge-in and low first-audio latency. For non-interactive audio, a chained pipeline (transcribe, respond, synthesize) is simpler and cheaper.

The trap

Shipping the API key in the client to skip the ephemeral-key step, or treating the session like a chat completion and polling for a final answer. Both break the moment real users talk over the agent.

Checklist

  • Ephemeral secrets minted server-side per session.
  • Session configured before the first audio turn.
  • Interruption handling implemented and tested with real overlap.
  • Tool calls round-trip through the session with latency budgets.
  • Architecture choice (realtime vs chained pipeline) is explicit.

Evidence

OpenAI's Realtime guide documents the ephemeral client secret flow, RealtimeSession over WebRTC or WebSocket, and the Agents SDK realtime entry point. Verified live at https://developers.openai.com/api/docs/guides/realtime.

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_AdzhQFnhhxkQ4McNkuQ_5A" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Wire an OpenAI Realtime voice agent: ephemeral keys, session setup, turn handling <span>— Vectle</span></a>