Real-Time Voice AI Agent Frameworks in Production: Comparing LiveKit Agents, Pipecat, OpenAI Realtime API, and Ultravox

Deploying conversational voice agents in production requires solving a fundamental physics and networking problem: human conversational turn-taking occurs within an average gap of 200 to 300 milliseconds. When an artificial conversational agent exceeds 600 to 800 milliseconds of round-trip latency, users perceive the interaction as sluggish, talk over the assistant, or experience severe turn collisions.

Historically, conversational voice systems relied on sequential request-response architectures: buffering an entire spoken turn to disk, sending the WAV payload to a Speech-to-Text (STT) API, awaiting a full textual completion from a Large Language Model (LLM), and finally streaming the text to a Text-to-Speech (TTS) engine. This cascaded batch approach routinely yielded 1,500 to 3,500 milliseconds of latency.

Modern real-time voice architectures eliminate batch boundaries through streaming token pipelines, low-latency audio transports, neural voice activity detection (VAD), and predictive turn-taking models. Engineering teams face critical architectural choices: whether to build modular cascaded pipelines (STT -> LLM -> TTS) or adopt native speech-to-speech (S2S) multimodal models, whether to transport media over WebSockets or WebRTC, and which orchestration runtime to standardize on.

This analysis examines the operational mechanics of real-time voice agents, details the underlying transport and turn-taking algorithms, benchmarks latency budgets, and compares four primary production solutions: LiveKit Agents, Pipecat, the OpenAI Realtime API, and Fixie AI's open-weight Ultravox.

Architectural Paradigms: Cascaded Pipelines vs. Native Speech-to-Speech

Real-time voice agent systems fall into two distinct architectural paradigms:

1. Cascaded (Modular) STT-LLM-TTS Pipelines

In a cascaded architecture, three specialized models execute concurrently in an asynchronous stream:

1. Streaming Speech-to-Text (ASR): Ingests raw audio frames (typically 16 kHz or 24 kHz PCM) over UDP or WebSockets and emits streaming partial transcripts with latency between 100 and 250 milliseconds.
2. Streaming LLM Generation: Ingests incoming text tokens and outputs completion tokens autoregressively. Time-to-First-Token (TTFT) for modern frontier models (such as Claude 3.5 Sonnet, GPT-4o-mini, or Llama 3.3 70B on high-throughput inference engines like vLLM or SGLang) ranges from 120 to 300 milliseconds.
3. Streaming Text-to-Speech (TTS): Consumes LLM token streams, buffers them into phonetically coherent clauses or sentences, and synthesizes continuous audio chunks. Modern neural TTS engines (such as Cartesia Sonic, ElevenLabs Flash v2.5, or Deepgram Aura) deliver a Time-to-First-Audio (TTFA) of 80 to 150 milliseconds.

User Audio (PCM)
       │
       ▼
┌───────────────────────────┐
│ Streaming STT (150ms)     │ ─── Partial Transcripts
└─────────────┬─────────────┘
              │ Text Stream
              ▼
┌───────────────────────────┐
│ Streaming LLM (180ms TTFT)│ ─── Token Stream (Sentence Boundary Chunker)
└─────────────┬─────────────┘
              │ Text Chunks
              ▼
┌───────────────────────────┐
│ Streaming TTS (100ms TTFA)│ ─── Synthesized Audio Packets
└─────────────┬─────────────┘
              │
              ▼
Agent Audio Output (Total Latency: ~430-650ms)

Key Advantages:

• Modularity and Model Swapping: Each component can be upgraded independently. A developer can switch from OpenAI to an open-weight Llama 3.3 model or swap TTS voices without refactoring agent logic.
• Granular Guardrails and Tool Calling: Text representations pass through deterministic regex filters, structured output validators, and MCP tool routers prior to audio synthesis.
• Cost Efficiency: Inference costs can be tightly optimized by selecting smaller language models or self-hosting open-weight components.

Key Limitations:

• Acoustic Information Loss: Transcribing speech to text discards all non-verbal acoustic cues: emotional tone, prosody, cadence, sarcasm, whispering, laughter, and background context.
• Compounded Pipeline Latency: Network hops between independent STT, LLM, and TTS providers add serialization and transport overhead.

2. Native Speech-to-Speech (S2S) Multimodal Architectures

Native S2S models (such as OpenAI GPT-4o Realtime, Moshi, and Gemini Live) process continuous audio tokens directly through a unified multimodal transformer backbone. Audio features are extracted via neural audio codecs (such as EnCodec, Mimi, or Mel-spectrogram encoders) and projected into the model's embedding space without intermediate text conversion.

Key Advantages:

• Acoustic Nuance Retention: The model natively comprehends user inflection, hesitations, emotional distress, and environmental sounds, modulating its synthesized vocal prosody in response.
• Theoretical Latency Floor: By bypassing independent pipeline boundaries, end-to-end latency can drop to 250 to 400 milliseconds.

Key Limitations:

• Provider Lock-In and Cost: Proprietary S2S APIs charge substantial premiums for audio token ingestion and generation.
• Constrained Observability: Inspecting intermediate reasoning, filtering hallucinations mid-stream, or running deterministic safety guardrails before audio synthesis is significantly more challenging.

Transport Layers: WebSockets vs. WebRTC vs. Telephony

The choice of network transport dictates minimum delivery latencies, packet loss tolerance, and client interoperability.

| Feature | Raw WebSockets (TCP) | WebRTC (UDP / RTP) | SIP / PSTN Trunking | | :--- | :--- | :--- | :--- | | Underlying Protocol | TCP / TLS | UDP / SRTP | UDP / TCP (SIP Signaling) + RTP | | Head-of-Line Blocking | Vulnerable (retransmits dropped packets) | Immune (drops late frames cleanly) | Immune (real-time media channel) | | Jitter Buffer Handling | Application-managed | Native browser/client engine | Gateway / PBX managed | | Acoustic Echo Cancellation | Manual implementation required | Browser/OS native hardware hooks | Hardware / DSP required | | Typical Network Overhead | 50 - 150 ms | 15 - 40 ms | 30 - 80 ms | | Target Clients | Server-to-server, simple web apps | Web apps, iOS/Android mobile, desktop | Twilio, Telnyx, enterprise call centers |

Why WebRTC Outperforms WebSockets for Production Voice

WebSockets operate over TCP. When a network packet is dropped over an unstable cellular or Wi-Fi connection, TCP forces the receiving socket to pause execution while retransmitting the lost segment (Head-of-Line blocking). In a voice stream, retransmitting a 20-millisecond audio packet that is already 300 milliseconds old is useless; it introduces audible stutter and delays all subsequent frames.

WebRTC uses RTP (Real-Time Transport Protocol) over UDP. When a packet is lost, the WebRTC jitter buffer applies packet loss concealment (PLC) or skips the missing frame entirely. WebRTC also provides standardized Interactive Connectivity Establishment (ICE), STUN/TURN NAT traversal, dynamic bandwidth estimation, and direct access to browser hardware-level Acoustic Echo Cancellation (AEC).

For telephony use cases, systems integrate SIP trunks (via Twilio, Telnyx, or Vonage) bridging PSTN audio into internal WebRTC Selective Forwarding Units (SFUs) or RTP media servers.

Voice Activity Detection, Turn-Taking, and Interruption Handling

The most difficult aspect of voice agent engineering is knowing when the human has finished speaking and when the human is interrupting the bot.

User: "Can you check my balance... [250ms pause] ...for my savings account?"
      │                     │
      │                     ▼
      │           Static VAD triggers here (False Interruption / Premature Cutoff)
      ▼
Semantic Turn Detector: Evaluates grammar/prosody -> Waits for completion -> Fires after "account"

1. Voice Activity Detection (VAD)

Traditional energy-based VAD calculates root-mean-square (RMS) decibel thresholds across audio buffers. In noisy real-world environments, background noise, throat-clearing, or keyboard typing constantly triggers false positives.

Modern production frameworks deploy lightweight neural VADs, primarily Silero VAD. Silero VAD operates on 30-millisecond audio chunks (512 samples at 16 kHz), computing speech probability via an ONNX-quantized recurrent network in less than 1 millisecond on CPU.

2. The Turn-Taking Dilemma and Semantic End-of-Utterance (EOU)

A pure VAD model only detects the presence or absence of speech energy. If an agent triggers whenever VAD detects 300 milliseconds of silence, it will constantly interrupt users who pause mid-sentence to think. Conversely, setting a static silence timeout of 800 to 1,000 milliseconds creates an unnatural, sluggish conversational lag.

Production systems solve this using two-stage turn detection:

1. Stage 1 (Acoustic Trigger): Silero VAD flags user silence exceeding a short threshold (e.g., 200 milliseconds).
2. Stage 2 (Semantic / Prosodic Evaluation): A dedicated turn-detector model analyzes the partial transcript and acoustic context to predict whether the utterance is semantically complete.

• Pipecat Smart Turn v3.2: An open-source, audio-native turn-detection transformer that analyzes acoustic prosody and falling pitch contours over the final seconds of audio.
• LiveKit Turn Detector (EOU): An end-of-utterance language classification model that evaluates grammatical completeness before releasing the turn to the LLM.
• Inworld Semantic VAD: A server-side model integrating acoustic signals with linguistic context.

3. Barge-In Mechanics and State Machine Cancellation

When an agent is actively speaking (streaming TTS audio to the client) and the user begins talking, the agent must yield immediately. Handling barge-in cleanly requires coordinated actions across the entire pipeline:

1. VAD Interruption Detection: The client or server VAD detects user speech exceeding an interruption confidence threshold (e.g., 3 consecutive speech frames / 90ms).
2. Audio Buffer Flush: The media transport immediately emits a clear/flush packet to empty the client-side jitter buffer and stop playback instantly.
3. Async Generation Cancellation: The orchestrator sends a cancellation token to abort the active LLM generation stream and stops streaming audio chunks from the TTS service.
4. Context Reconciliation: The agent state machine records what portion of the assistant's response was actually played to the user prior to the interruption, appending only the heard text to the conversation history before processing the new user query.

Production Framework Comparison

Four architectures dominate modern production deployments:

| Feature / Metric | LiveKit Agents | Pipecat (Daily) | OpenAI Realtime API | Ultravox (Fixie AI) | | :--- | :--- | :--- | :--- | :--- | | Architecture Type | Room-based WebRTC worker | Composable frame pipeline | Managed multimodal S2S | Open-weight multimodal S2S | | Transport Support | WebRTC (Native SFU), SIP | WebRTC (Daily), WebSockets, SIP | WebSockets, WebRTC | WebSockets, REST | | Language / Runtime | Python, Node.js, Go | Python | REST / WebSocket / WebRTC | Python / PyTorch / vLLM | | VAD / Turn Detection | Silero VAD + LiveKit EOU | Silero VAD + Smart Turn v3.2 | Built-in Server VAD / Turn | End-to-end token generation | | Ecosystem Integrations | Deepgram, Cartesia, ElevenLabs, OpenAI, Anthropic | 60+ modular service providers | OpenAI ecosystem only | Any Hugging Face / vLLM backend | | Barge-in Latency | < 100 ms (room-level flush) | < 100 ms (frame cancel) | Built-in server interruption | Pipeline-dependent | | Deployment Model | Self-hosted or LiveKit Cloud | Self-hosted or Pipecat Cloud | Managed OpenAI API | Self-hosted GPU or Fixie Cloud | | Open Source | Apache 2.0 | BSD-2-Clause | Proprietary | Apache 2.0 |

LiveKit Agents: Worker Process connects to WebRTC SFU Room as a peer participant.
┌────────────────────────────────────────────────────────┐
│ LiveKit WebRTC Server (SFU)                            │
│   ▲                      ▲                             │
│   │ WebRTC Tracks        │ WebRTC Tracks               │
│   ▼                      ▼                             │
│ Client (Browser/App)   Agent Worker (Python SDK)       │
│                        ├── Silero VAD + EOU Classifier │
│                        ├── STT Plugin (Deepgram)       │
│                        ├── LLM Plugin (Claude/GPT-4o)  │
│                        └── TTS Plugin (Cartesia)       │
└────────────────────────────────────────────────────────┘

Pipecat: Modular Frame Processor Pipeline.
┌──────────────────────────────────────────────────────────────────────────┐
│ Transport In ──► VAD Analyzer ──► STT Service ──► Context Aggregator ──┐ │
│                                                                        │ │
│ Output Transport ◄── TTS Service ◄── LLM Service ◄─────────────────────┘ │
└──────────────────────────────────────────────────────────────────────────┘

1. LiveKit Agents

LiveKit Agents is built on top of LiveKit's open-source WebRTC media server. Instead of running standalone HTTP servers, voice agents run as worker processes that join WebRTC rooms as active participants.

Core Strengths:

• Infrastructure Maturity: Inherits battle-tested WebRTC clustering, automatic load balancing across worker pools, and sub-30ms global media routing.
• Native Multi-Modal Support: An agent can consume audio, publish video, subscribe to screen shares, and receive data channels within the same room.
• Turn-Taking Stack: Integrates bundled inference for Silero VAD alongside LiveKit's End-of-Utterance (EOU) model, handling dynamic silence windows out of the box.

2. Pipecat

Pipecat is an open-source Python framework developed by Daily. It structures voice workflows as a directed pipeline of asynchronous frame processors.

Core Strengths:

• Architectural Modularity: Audio, text, and control signals flow through discrete frames (AudioFrame, TextFrame, InterruptionFrame). Developers can insert custom filters, translation layers, or observability hooks anywhere in the pipeline.
• Transport Agnosticism: Runs seamlessly across Daily WebRTC, raw WebSockets, local microphone/speaker audio, and telephone SIP bridges.
• Smart Turn Integration: Natively integrates the Smart Turn v3.2 model to minimize false interruptions without sacrificing response speed.

3. OpenAI Realtime API

The OpenAI Realtime API exposes GPT-4o's multimodal capabilities over persistent WebSockets and WebRTC connections.

Core Strengths:

• Minimal Infrastructure Code: Handles audio input tokenization, VAD, turn-taking, tool execution, and voice output synthesis within a single API connection.
• Expressive Audio Quality: Capable of modulating vocal pitch, cadence, and emotion based on contextual prompts and system instructions.
• Integrated Tool Calling: Supports streaming function calls directly over the audio session while maintaining conversational context.

4. Ultravox (Fixie AI)

Ultravox is an open-weight multimodal Speech-Language Model that connects a speech encoder (such as Whisper or WavLM) directly into an open-weight LLM backbone (such as Llama 3.3 or Mistral) via a trained multimodal projection layer.

Core Strengths:

• Open Weights and Sovereign Deployment: Entirely self-hostable on internal GPU infrastructure, eliminating third-party API dependencies and data privacy risks.
• Fast Time-to-First-Token: Processes speech embeddings directly into the language model, bypassing external STT service network hops and yielding initial text generation within 150 milliseconds.

Latency Breakdown and Budgeting (Time-to-First-Audio)

Achieving conversational fluidity requires tuning every stage of the pipeline. The table below details realistic latency budgets for optimized production systems:

| Pipeline Stage | Component / Provider | Typical Latency (P50) | Latency (P95) | Optimization Techniques | | :--- | :--- | :--- | :--- | :--- | | Audio Ingest & Network | WebRTC UDP / Opus codec | 25 ms | 50 ms | Regional SFU placement, UDP jitter tuning | | VAD + Turn Detection | Silero VAD + Smart Turn v3.2 | 180 ms | 250 ms | Speculative turn commit, early endpointing | | Speech-to-Text (STT) | Deepgram Nova-3 / AssemblyAI | 140 ms | 220 ms | Streaming interim transcripts, chunked ASR | | LLM Time-to-First-Token | Claude 3.5 Sonnet / Llama 3.3 (vLLM) | 160 ms | 280 ms | Prompt caching, prefix prefilling, FP8 compute | | Sentence Chunker Gate | Regex boundary / Punctuation buffer | 40 ms | 80 ms | First-chunk punctuation heuristics (comma/clause) | | Text-to-Speech (TTS) | Cartesia Sonic / ElevenLabs Flash | 90 ms | 150 ms | Streaming PCM chunks, 24 kHz raw output | | Audio Egress & Network | WebRTC Playback Buffer | 25 ms | 50 ms | Low client-side buffer depth (20-40ms) | | Total Round-Trip (TTFA) | Optimized Cascaded Pipeline | 660 ms | 1,080 ms | Parallelized streaming across all stages | | Total Round-Trip (TTFA) | Native S2S (OpenAI Realtime) | 380 ms | 620 ms | Direct speech-to-speech token stream |

Serving Economics and Cost Comparison

Voice agent operating costs scale with active conversational minutes rather than isolated token requests.

| Architectural Stack | Component Breakdown | Estimated Cost / Audio Minute | 1,000 Concurrent Calls (Monthly) | | :--- | :--- | :--- | :--- | | Proprietary Multimodal S2S<br>(OpenAI Realtime API) | Audio In ($0.06/min) + Audio Out ($0.24/min) + Text tokens | $0.12 - $0.30 / min | $51,840 - $129,600 (at 40 hrs/mo/seat) | | Managed Cascaded Stack<br>(Deepgram + GPT-4o-mini + Cartesia) | STT ($0.0043/min) + LLM ($0.002/min) + TTS ($0.025/min) + WebRTC ($0.004/min) | $0.035 - $0.055 / min | $15,120 - $23,760 | | Hybrid Open/Managed Stack<br>(Deepgram + Llama 3.3 70B vLLM + Cartesia) | STT ($0.0043/min) + GPU LLM ($0.006/min) + TTS ($0.025/min) + LiveKit Cloud | $0.038 - $0.050 / min | $16,410 - $21,600 | | Fully Self-Hosted Open Stack<br>(Whisper v3 Turbo + Llama 3.3 70B + Kokoro/Piper) | 2x NVIDIA H100 80GB nodes (~$6.00/hr total supporting ~120 concurrent streams) | $0.008 - $0.015 / min | $3,450 - $6,480 (plus fixed infra costs) |

For enterprise deployments operating at high call volumes (such as customer support contact centers processing millions of minutes monthly), cascaded and self-hosted pipelines offer 70% to 90% cost savings compared to proprietary speech-to-speech APIs, while preserving total control over tool execution, privacy compliance, and latency tuning.

Architectural Decision Framework

When selecting a framework for production voice agents:

1. Choose LiveKit Agents when building complex multi-user WebRTC rooms, video-enabled agents, or applications requiring enterprise-grade media routing, global SFU scaling, and built-in dispatch workers.
2. Choose Pipecat when building custom, flexible pipeline architectures across diverse transports (PSTN telephony, raw WebSockets, Daily WebRTC) with granular frame-level control and multi-provider orchestration.
3. Choose OpenAI Realtime API when rapid time-to-market and expressive vocal prosody outweigh per-minute API costs, and when the application does not require self-hosted compliance or multi-vendor redundancy.
4. Choose Ultravox when data residency, zero third-party egress, and self-hosted open-weight execution on private GPU clusters are mandatory requirements.

Sources

• LiveKit Agents Documentation and Architecture
• Pipecat Open-Source Framework Documentation
• Pipecat Smart Turn v3.2 Model Repository
• OpenAI Realtime API Guide and Specifications
• Fixie AI Ultravox Repository and Architecture
• Silero VAD: Pre-trained Enterprise-Grade Voice Activity Detector
• Building Enterprise Realtime Voice Agents from Scratch (arXiv:2603.05413)