A voice agent can get every word right and still handle a call badly. It talks over you, waits too long, or misses that you sounded unsure. On Monday morning, October 19, a post from Naman Jain put a name to the cause and drew wide discussion on X: "I think the biggest gap in voice AI is that agents make decisions from a transcript, not from the audio." His write-up is titled Voice agents aren't listening. They're reading.
That points to an architecture choice every voice product makes. Speech can pass through a chain of separate models that turn it into text and back, or it can go to a model that works with the audio itself. This explainer covers how each design works, what a transcript leaves out, why the chained design is still in wide use, and how to pick one. It uses OpenAI's developer documentation as the main reference, because that documentation sets out the trade-offs plainly.
The three ways to build a voice agent
OpenAI's voice agents guide says the key design choice is "how speech connects to reasoning and tools." It lists three options.
1. Chained (cascaded) pipeline
This is the classic design. Your application runs three stages in order:
- Speech-to-text turns the caller's audio into a transcript.
- A text agent (an LLM with tools and business rules) reads the transcript and writes a reply.
- Text-to-speech reads the reply aloud.
OpenAI's migration guide uses both names: "A chained, or cascaded, voice pipeline adds speech-to-text before that agent and text-to-speech after it." This is also how ChatGPT's older voice mode worked. TechCrunch reported that the previous model "combined a speech-to-text model to transcribe speech, a large language model to generate responses, and a text-to-speech model to deliver the final answer."
2. Speech-to-speech
Here one model takes audio in and sends audio out. OpenAI's Realtime API guide says "the model works directly with audio, maintains conversation state, and can call tools." OpenAI calls it the best starting point "for voice agents that need barge-in, low first-audio latency, natural turn taking, and realtime tool use." Barge-in means the caller can cut in while the agent is talking.
3. Full duplex with delegation
This is the newest option. A voice model handles only the conversation and passes the hard work to a separate backend. OpenAI's version is GPT-Live, which it introduced in ChatGPT on July 8, 2026 and released in the API on September 10. Full duplex means it "can listen and speak at the same time," according to the GPT-Live getting started guide. The guide calls the hand-off to the backend "delegation." In OpenAI's words, GPT-Live "listens, speaks, and decides when to ask the backend for help," and the backend "reasons, uses tools, and returns results for GPT-Live to communicate."
What a transcript leaves out
A transcript records which words were said. It does not record how they were said, or the sounds around them. In a chained pipeline, the text agent that makes the decisions only gets the transcript. That is the gap Jain describes.
OpenAI's migration guide gives concrete cases where the words are not enough: "Some decisions require the sound itself, such as detecting a voicemail beep or recognizing a recorded greeting from its timing." Its checklist for moving an app to GPT-Live tells developers to "identify decisions that need the original sound, beyond the words in a transcript."
Timing is the other loss. A turn-based pipeline has to decide when the caller has finished before it can start the next stage. If it decides too early, it interrupts. If it waits too long, there is dead air. We compared the tools developers use for that decision in our explainer on voice activity detection and turn-taking.
OpenAI's own numbers show how much this matters. In its API launch post, OpenAI says GPT-Live-1 reasons "over incoming and outgoing audio together, avoiding the latency and brittle handoffs of chained" speech-to-text, LLM and text-to-speech architectures. It also cites a customer result: "in early evaluations, Speak found that GPT‑Live‑1 gave learners more time to think before the language tutor responded, cutting interruptions by almost 80% versus previous turn-based systems." That is a vendor-reported figure from one customer, not an independent benchmark.
The GPT-Live launch post describes the behavior a turn-based system finds hard: the model "can show it's paying attention with phrases like 'mhmm' or 'yeah', engage in quick back-and-forth, or just stay quiet when you need a moment to think."
Why chained pipelines are still everywhere
The chained design is not a mistake. It gives developers control that a single audio model does not. OpenAI's voice agents guide recommends it "when each stage needs to be visible or replaceable." Its example: "you might store the transcript, run policy checks before the text agent responds, call internal systems, then generate speech only after the workflow reaches an approved answer."
That last point matters for regulated work. The migration guide explains the trade-off directly: "A text agent or chained pipeline can validate a complete reply before displaying or speaking it. With GPT-Live, conversation and backend work run at the same time." If every spoken reply must pass a check first, the guide says to put that check "in the audio playback path your application controls." It also warns: "A corrective instruction cannot retract audio already heard."
Put simply, the chained pipeline can approve a reply before anyone hears it. A full-duplex agent talks while it works, so any check has to sit between the model and the speaker.
Other reasons teams keep a chain:
- Each stage can be swapped on its own. The guide says you can "replace each component independently."
- The guide says you can "inspect or transform intermediate text" between stages.
- An existing text agent can stay in place. GPT-Live also allows this, as the next section shows.
The middle ground: a voice front end and a text back end
GPT-Live keeps part of the chained idea. The voice model handles the talking, and a text model does the thinking. At launch in ChatGPT, OpenAI said GPT-Live "will use GPT‑5.5 in the background." In the API, the backend can be "a backend text model like GPT‑6 Astra or a third-party model," per the API post.
The GPT-Live guide offers two delegation modes. With Responses delegation, OpenAI runs a hosted backend model. With client delegation, you "connect any model, agent harness, or service your application runs." For teams with a chained pipeline, the migration guide says GPT-Live "replaces the separate speech-to-text and text-to-speech stages," and the agent, tools and state stay in the backend.
There is a catch that ties back to Jain's point. The backend may still work from text. The migration guide says "GPT-Live hears the call, but delegation does not automatically send audio to your backend." In client mode, your application "supplies the request context and any audio the backend needs." So a full-duplex front end fixes the timing problem in the conversation itself. It does not, on its own, give the decision-making backend the sound of the call. For something like answering-machine detection, the guide suggests sending a copy of the call audio to a separate audio-capable detector.
How to choose
Based on OpenAI's own guidance, the choice comes down to what you need most:
- Pick a chained pipeline if every reply must be checked before it is spoken, if you need a stored transcript at each step, or if you want to replace each stage on its own.
- Pick speech-to-speech (OpenAI's Realtime API) if you want one model to "interpret audio, decide what to do, and respond in speech" in a single session, with barge-in and low first-audio latency.
- Pick full duplex with delegation (GPT-Live) if you want natural turn-taking and interruptions but also want to keep an existing text agent, or choose its model separately.
For consumers, the same split shows up in phone assistants. We compared the current options in Siri vs Gemini vs ChatGPT Voice.
What it costs
As of October 2026, OpenAI's GPT-Live-1 model page says "voice sessions cost $0.05 per minute, billed per second. Backend model and tool usage is billed separately." The API post describes the $0.05 per minute rate as the price "for the front-end voice layer." A chained pipeline's cost depends on the speech-to-text, LLM and text-to-speech services you choose. For voice service prices more broadly, see our AI voice and music pricing comparison.
How to test which design works for you
OpenAI's voice agents guide recommends testing the conversation and the finished task separately. For GPT-Live it lists four things to measure on their own:
- Task and tool outcomes: did the right action happen, and does the spoken confirmation match it.
- Conversational timing: response timing, "unwanted silence, overlap, and yielding to interruptions."
- Speech and language: recognition "across accents, background noise, language switches, names, and numbers."
- Session reliability: dropped audio, timeouts and failed connections.
It also suggests adding difficulty in three stages it calls Crawl, Walk and Run: synthetic speech for fixed single-turn tests, then real human recordings, then a simulated caller for full multi-turn calls with interruptions. And it adds that automated scores should be paired with human listening "to assess pronunciation, naturalness, and whether the conversation feels appropriately paced."
If you are building on open models instead, our explainer on Ant Group's Ming-3.0-audio models covers one recent open release for speech.
Why it matters
Voice agents now take real phone calls. monday.com's Harmony is one example of voice AI agents that call sales leads. The architecture decides what the agent can notice. A chained pipeline is easier to audit, but it decides from words alone and has to guess when you are done talking. Audio-native and full-duplex models handle timing better, by OpenAI's account, but they talk while they work, which moves safety checks into the audio path. Neither design gives a text backend the sound of the call unless the developer routes it there.
Sources
- Naman Jain on X: "Voice agents aren't listening. They're reading." (October 19, 2026)
- OpenAI: Voice agents guide
- OpenAI: Getting started with GPT-Live
- OpenAI: Migrate to GPT-Live
- OpenAI: Getting started with the Realtime API
- OpenAI: GPT-Live-1 model page and pricing
- OpenAI: Introducing GPT-Live (July 8, 2026)
- OpenAI: Build more natural voice experiences with GPT-Live-1 in the API (September 10, 2026)
- TechCrunch: OpenAI releases new voice models for more natural live conversations
