Engineers Justin Uberti and Zahan Malkani explain how a new realtime voice architecture, GPT‑Live, was developed in six months to reduce conversational latency and keep audio flowing end to end. The system places a full‑duplex voice model on the live path so it can listen and speak simultaneously, while heavier reasoning or tool use is handled asynchronously by frontier models such as GPT‑5.5.
Architecture for continuous inference and state
The team moved away from turn‑based designs that relied on a turn detector and cascaded speech‑to‑text/LLM/text‑to‑speech stages. Instead, GPT‑Live streams incoming audio directly into the voice model and streams outbound audio back to the user, separating the real‑time media path from application logic and delegation. According to the authors, this keeps the live path small and predictable and prevents slow backend calls from interrupting audio.
To support long sessions, the system implements stateful inference with managed handoffs. Replacement model instances are warmed and prefilled with session context, run in parallel with the active instance, and cut over only when ready. The same technique supports dynamic context compaction and avoids blocking the live media path when the conversation’s context must be reduced and the KV cache rebuilt.
Delegation, transport, and startup optimizations
Delegation to frontier models is designed to be fast enough to feel natural: inference sessions for the frontier model are prefetched and kept available, with prompt caching and session affinity to reduce latency. The application server also converts continuous speech into discrete user and assistant turns for other systems, maintaining a speculative view for UI updates and an authoritative record for logging.
On the transport side, the media frontend and inference logic were rewritten in Go, replacing a Python asyncio implementation; the authors report that the new p95 latency matches the previous p50. WebRTC remains the transport foundation, and the team introduced the WebRTC Abridged Roundtrip Protocol (WARP) to reduce startup from six round trips to one via techniques including SPED, DTLS 1.3 handshakes, SNAP, and data‑channel pre‑negotiation instead of DCEP. Instant Connect pre‑negotiates signaling so a session can begin with a single UDP packet.
The authors note that WARP support has already been added to libwebrtc and Pion and that the proposals are being advanced through IETF’s TSVWG. Before public rollout, the system was exercised by shadowing a fraction of production ChatGPT Voice sessions in read‑only mode to observe real client and network behavior.
Original source: OpenAI News