Real-Time Voice Agent Architecture: WebRTC, Cascaded Pipelines vs. Native Speech-to-Speech, and Sub-500ms Latency Budgets

Building production-grade real-time voice AI systems requires engineering around a strict physical constraint: human conversational cadence. In natural human dialogue, the typical gap between turns ranges from 200 to 300 milliseconds. When an interactive voice agent incurs a total round-trip latency above 700 milliseconds, users perceive the interaction as sluggish. When latency exceeds 1,000 milliseconds, conversational dynamics collapse into frequent interruptions, speech collisions, and awkward silence.

Achieving sub-second response times demands purpose-built media infrastructure, streaming data pipelines, and optimized model inference. Modern voice architectures diverge into two primary paradigms: the modular cascaded pipeline (Voice Activity Detection, Speech-to-Text, Language Model, and Text-to-Speech) and native multimodal speech-to-speech models.

Real-time voice agent architecture comparing cascaded pipelines with native speech-to-speech models

Media Transport: Why WebRTC Outperforms WebSockets

Traditional web applications rely on HTTP/REST or TCP-based WebSockets for client-server communication. For real-time bi-directional voice streaming, standard WebSockets introduce severe latency vulnerabilities:

Head-of-Line Blocking: Because TCP enforces strict packet ordering and reliable delivery via retransmissions, a single dropped audio packet halts all subsequent packet processing until the missing frame is recovered.
Lack of Media-Aware Jitter Buffering: WebSockets treat audio as generic byte streams without RTP (Real-time Transport Protocol) packet sequence numbers or audio timestamp synchronization.
Missing Acoustic Feedback Controls: Browser and mobile audio pipelines require hardware-integrated Acoustic Echo Cancellation (AEC) and automatic gain control to prevent the speaker output from bleeding back into the microphone.

Production voice agent systems use WebRTC (RFC 8825) over UDP, orchestrated by Selective Forwarding Units (SFUs) such as LiveKit. WebRTC provides:

Transport latency under 50 milliseconds across global edge networks.
Packet Loss Concealment (PLC) and adaptive bitrate Opus codec negotiation that gracefully degrades audio quality without blocking audio playback.
Native browser and OS audio track bindings with hardware echo cancellation.
Bidirectional data channels for passing out-of-band JSON metadata, transcripts, and interruption signals synchronously with audio media tracks.

The Cascaded Pipeline: Modular Inference

The cascaded architecture decomposes the conversational loop into discrete, specialized services executed over streaming connections.

1. Audio Framing and Voice Activity Detection (VAD)

The system must detect speech boundaries without waiting for user-initiated signals. Modern implementations run neural VAD models such as Silero VAD on client devices or directly in the SFU worker process:

Silero VAD evaluates 30-millisecond audio frames in less than 1 millisecond on a single CPU thread.
The system tracks speech probability thresholds (typically 0.5) and calibrates an end-of-speech silence window (typically 200 ms to 350 ms).
Premature turn cutoffs are mitigated using semantic endpointing heuristics in conjunction with speech-to-text token confidence.

2. Streaming Speech-to-Text (STT)

Once VAD detects voice activity, raw PCM/Opus audio is streamed via WebSocket to a specialized automatic speech recognition service such as Deepgram Nova-3 or AssemblyAI:

Time-to-First-Transcript (TTFT) averages 100 ms to 200 ms.
The STT engine emits interim partial transcripts to prime downstream prompt buffers, finalizing the utterance with timestamped word alignments upon endpoint detection.

3. Fast Language Model Inference

Upon receiving the finalized user transcript, the agent controller dispatches the context to a high-throughput LLM. For production voice applications, teams prioritize low Time-to-First-Token (TTFT) over parameter scale:

Models such as Claude 3.5 Haiku, Llama 3.3 70B (served via TensorRT-LLM or vLLM), and GPT-4o-mini provide initial tokens within 200 ms to 350 ms.
Prompt caching reduces prefill latency on multi-turn conversations by storing the static system instructions and preceding dialogue context in GPU memory.

4. Streaming Text-to-Speech (TTS)

Rather than waiting for the complete LLM response, the agent streams generated tokens directly to a streaming TTS engine. State Space Model (SSM) architectures such as Cartesia Sonic achieve a Time-to-First-Audio (TTFA) of 40 ms to 90 ms:

The TTS worker buffers tokens until the first syntactic boundary (a comma, period, question mark, or semicolon) is detected.
Synthesized audio chunks are immediately packetized into Opus RTP frames and broadcast over the WebRTC audio track to the user.

Cumulative Latency Budget (Cascaded)

Audio Transport (WebRTC): 30 ms to 50 ms
Turn Detection / Silence Window: 200 ms to 300 ms
Streaming STT Interim Finalization: 100 ms to 150 ms
LLM Time-to-First-Token: 200 ms to 300 ms
Token Clause Buffering: 40 ms to 80 ms
TTS Time-to-First-Audio: 40 ms to 100 ms
Total Perceived Turn Latency: 610 ms to 980 ms

Native Speech-to-Speech (S2S) Architecture

Native multimodal speech-to-speech models, exemplified by the OpenAI Realtime API and Google Gemini Multimodal Live, eliminate intermediate text representations.

In this architecture, raw audio tokens enter the transformer directly, and the neural network autoregressively produces output audio tokens:

Latency Optimization: By eliminating discrete STT tokenization and TTS generation stages, speech-to-speech models compress perceived end-to-end response latency to 300 ms to 500 ms.
Paralinguistic Nuance: Native models preserve acoustic context, including vocal inflection, sarcasm, whispers, laughter, emotional tone, and non-native accents, which are lost when converting speech to plain text.
Simplified Orchestration: Developers maintain a single WebSocket or WebRTC connection to the foundation model provider rather than orchestrating three separate microservices.

Engineering Trade-Offs of Native S2S

Despite latency advantages, native speech-to-speech systems introduce distinct production challenges:

Inference Economics: Audio token pricing is substantially higher than text token pricing. The OpenAI Realtime API prices audio input at approximately $0.06 per minute ($40 to $100 per 1M tokens) and audio output at $0.24 per minute ($80 to $200 per 1M tokens). When full conversation history is replayed across multi-turn sessions, un-cached sessions can reach $0.20 to $0.45 per minute.
Opaque Guardrails: In a cascaded pipeline, security guardrails, PII filters, and deterministic intent classifiers inspect the plain text transcript before the LLM generates a response. With end-to-end S2S, filtering must occur on raw audio or after generation has begun.
Vendor Lock-In: Deployments become tightly coupled to a single proprietary foundation model provider's pricing, availability, and voice catalog.
Tool Calling Latency: When an S2S model executes an external tool or database query, the audio generation pauses, forfeiting the native streaming latency benefit.

Critical Systems Engineering Patterns

Operating voice agents in production requires solving real-time synchronization and concurrency problems.

1. Interruption Handling (Barge-In)

When a user speaks while the agent is generating audio, the system must perform immediate cancellation:

Client-side or edge VAD detects user speech energy exceeding a calibrated threshold.
The client emits an immediate cancellation signal over the WebRTC data channel.
The server cancels in-flight LLM generation and aborts downstream TTS synthesis jobs.
The client flushes its local audio playback buffer within 100 ms to 150 ms to silence the agent instantly.
The conversational context manager truncates the assistant transcript to match the exact audio timestamp played before the interruption occurred.

2. Sentence and Clause Chunking

Feeding single tokens to TTS produces disjointed, unnatural prosody, while waiting for full sentences introduces 500 ms of unnecessary latency. Production agents apply regex-based token stream chunkers:

First chunk threshold: Emitted after 4 to 8 words or upon hitting the first comma, colon, or dash, minimizing Time-to-First-Audio.
Subsequent chunks: Emitted on complete sentence terminators (., !, ?) to allow the TTS engine to infer appropriate pitch declination and intonation.

3. Hardware vs. Software Echo Cancellation

If the user uses open speakers rather than headphones, the agent's synthesized voice output travels into the microphone. Without Acoustic Echo Cancellation (AEC):

The STT model transcribes the agent's own speech.
The VAD interprets the agent's audio as a user barge-in, causing the system to interrupt itself.
Production systems enforce WebRTC browser AEC or embed DSP algorithms (such as WebRTC AEC3 or SpeexDSP) on edge hardware.

Architecture Comparison

Cascaded Pipeline (LiveKit + Deepgram + Claude/Llama + Cartesia)

Latency Floor: 550 ms to 800 ms
Cost Range: $0.03 to $0.12 per minute
Tool Calling Flexibility: High; full support for JSON schema tools, structured outputs, and intermediate validation
Guardrails and Redaction: High; deterministic text inspection before and after LLM generation
Voice Customization: High; multi-vendor voice selection, custom SSM voice cloning, and phoneme control
Vendor Independence: High; any STT, LLM, or TTS service can be swapped independently

Native Speech-to-Speech (OpenAI Realtime / Gemini Live)

Latency Floor: 300 ms to 500 ms
Cost Range: $0.15 to $0.45 per minute
Tool Calling Flexibility: Moderate; tool calling execution introduces stalls in audio streaming
Guardrails and Redaction: Low; limited visibility into intermediate representations
Voice Customization: Low; restricted to provider-hosted voice catalogs
Vendor Independence: Low; full proprietary lock-in to the model provider

Implementation Takeaways

For transactional, compliance-heavy, and high-volume enterprise deployments, cascaded pipelines remain the dominant architectural choice due to cost efficiency, granular guardrail enforcement, and component modularity.

For consumer applications, relationship companions, and high-interactivity scenarios where sub-400ms latency and emotional prosody dictate user retention, native speech-to-speech models provide a superior conversational feel at a premium operational cost.