Real-time voice AI agents depend on two sequential decisions: identifying when a user is speaking, and predicting when they have finished their conversational turn. Acoustic Voice Activity Detection (VAD) handles the first problem at the audio frame level, while turn-taking endpointing manages dialogue state transitions.
Mistakes in either layer degrade conversational flow. Inaccurate VAD causes false interruptions from background noise or misses quiet speech onsets. Naive silence-based endpointing creates an unresolvable trade-off: short silence thresholds cut users off mid-sentence, while conservative thresholds add hundreds of milliseconds of dead air to every interaction.
Acoustic Frame-Level VAD Architectures
Acoustic voice activity detectors process continuous PCM audio streams divided into short time slices, returning a binary speech decision or speech probability score for each frame.
WebRTC VAD
Google WebRTC VAD is a classical statistical detector implemented in C. It operates on 10 ms, 20 ms, or 30 ms PCM audio frames at sample rates of 8 kHz, 16 kHz, 32 kHz, or 48 kHz.
- Architecture: Gaussian Mixture Models (GMM) trained across six distinct sub-band energy features.
- Compute footprint: Extremely low memory footprint under 50 KB with sub-millisecond CPU execution time per frame.
- Operating modes: Four configurable aggressiveness modes (0 to 3) balancing sensitivity against false positives.
- Limitations: Highly sensitive to non-stationary acoustic noise, HVAC hum, keyboard clicks, breathing, and background media. Lacks temporal sequence modeling across multi-second utterances.
Silero VAD
Silero VAD is a lightweight neural network detector available in ONNX and TorchScript formats that has become the standard open-source VAD for streaming voice pipelines.
- Architecture: Compact convolutional and recurrent network (under 2 MB model size) trained on tens of thousands of hours of multi-lingual speech corpora and noise environments.
- Frame size: Native processing of 30 ms (512 samples at 16 kHz or 256 samples at 8 kHz) audio chunks.
- Compute footprint: Real-time factor (RTF) of approximately 0.004 on modern CPU cores, requiring less than 0.5% of a single CPU thread for real-time streaming.
- Stateful inference: Maintains hidden recurrent states across sequential audio frames to track acoustic context and speech continuity.
- Noise robustness: Effectively filters non-speech transients, background music, whispered speech, and room reverberation.
PyAnnote Audio Segmentation
PyAnnote.audio provides neural segmentation models based on the PyanNet architecture, primarily designed for speaker diarization and multi-speaker overlapping speech detection.
- Architecture: Trainable SincNet convolutional front-end filterbanks coupled with bidirectional LSTM layers and multi-layer perceptron projection heads.
- Window size: Evaluates sliding windows of 5.0 to 10.0 seconds with step intervals of 2.5 to 5.0 seconds.
- Capabilities: Outputs multi-class powerset probabilities, allowing simultaneous tracking of up to three overlapping speakers alongside binary voice activity.
- Serving trade-off: Significantly higher computational cost and latency than Silero VAD, making it ideal for offline transcription, post-call diarization, and multi-party meeting segmentation rather than ultra-low-latency real-time voice loops.
The Turn-Taking Problem and Endpointing
Voice activity detection classifies whether human speech is physically present in an audio frame. It does not determine whether a speaker has concluded their thought.
User Speech: [ "I need to check my balance" ] --- [ 450ms pause ] --- [ "for account 402" ]
|
Silence VAD Trigger (300ms threshold): ----------------> Interruption & Premature QueryWhen systems rely strictly on VAD silence thresholds, engineering teams face a structural dilemma:
- Aggressive Silence Threshold (200 ms to 400 ms): Minimizes conversational latency, but prematurely cuts off callers who hesitate, search for words, or pause between clauses.
- Conservative Silence Threshold (1000 ms to 1500 ms): Prevents accidental cut-offs, but introduces noticeable dead air before the agent responds, breaking conversational realism.
Semantic and Multi-Modal Endpointing
Modern voice agent frameworks resolve the endpointing dilemma by layering semantic and acoustic turn detection models on top of baseline frame VAD.
Neural Audio Turn Detectors
Dedicated turn-taking models, such as the LiveKit Turn Detector and Pipecat Smart Turn, analyze acoustic prosody directly from incoming audio.
- Prosodic Feature Extraction: Analyzes pitch contours (falling fundamental frequency signaling declarative completion versus rising pitch signaling continuation or questions) and speaking rate.
- Dynamic Endpointing: Dynamically adjusts required silence duration based on speaker cadence, allowing instant responses when an utterance ends with falling terminal pitch while extending silence windows during mid-sentence pauses.
- Benchmarking: According to turn-taking evaluations on LiveKit eot-bench, neural turn models achieve cut-off rates below 5% at roughly 540 ms of post-speech latency, compared to over 1100 ms required by fixed silence timers to achieve equivalent accuracy.
Streaming ASR Endpointing
Integrated Speech-to-Text endpointing delegates turn boundaries to the streaming transcription engine, as seen in architectures like Deepgram Flux and AssemblyAI Universal-3 Pro.
- Joint Token and Acoustic Probability: Evaluates both acoustic silence and syntactic completeness of partial transcript tokens.
- Grammar-Aware Decisions: If the partial transcript ends on a preposition or incomplete clause (such as "I am flying to..."), the endpointer delays turn completion. When a grammatically complete intent is recognized, the system commits the turn immediately.
Interruption and Barge-In Pipelines
Low-latency turn taking requires robust barge-in handling to allow users to interrupt an agent mid-sentence:
- Acoustic Echo Cancellation (AEC): Filters the agent's outgoing TTS audio playback from the incoming microphone feed.
- Streaming VAD Trigger: An active Silero VAD instance monitors the incoming stream during playback. Upon detecting sustained speech onset (typically 100 ms to 150 ms of speech frames), the system emits an interruption event.
- Playback Flush: The pipeline immediately halts downstream TTS generation, flushes client-side audio playback buffers, and rolls back dialogue state to process the new user utterance.
Architectural Trade-offs Across Engines
Key architectural characteristics across voice detection and endpointing engines include:
- WebRTC VAD: GMM energy filter with sub-50 KB memory footprint. Processes 10 ms to 30 ms frames with sub-0.001 CPU RTF. Designed for raw energy gating in embedded environments without turn awareness.
- Silero VAD v5: Neural CNN-RNN architecture with a 1.6 MB to 2.2 MB memory footprint. Processes 30 ms frames (512 samples) with approximately 0.004 CPU RTF. Designed for streaming speech gating and barge-in detection.
- PyAnnote Audio 3.1: SincNet and BiLSTM neural segmentation model with a 5.7 MB footprint. Processes 5.0 s to 10.0 s sliding windows with 0.05 to 0.20 CPU RTF. Designed for multi-speaker diarization and overlapping speech detection.
- Neural Turn Detectors: Multi-modal prosodic and acoustic classifiers with a 10 MB to 50 MB memory footprint. Evaluates 250 ms to 1.0 s temporal contexts with 0.01 to 0.03 CPU RTF. Designed specifically for turn boundary prediction and syntactic completion.
Engineering Recommendations for Production Voice AI
- Deploy Silero VAD for Frame-Level Gating: Use Silero VAD as the default streaming audio filter and barge-in detector. Its 2 MB footprint and 0.004 RTF allow local execution on edge clients or audio ingress servers without measurable CPU overhead.
- Decouple Interruption Detection from Turn Completion: Use low-latency frame VAD (100 ms onset window) strictly for triggering TTS playback interruptions, but route end-of-utterance decisions through an audio-linguistic turn detector or streaming ASR endpointer.
- Avoid Fixed Silence Thresholds Below 800 ms Without Model Gating: If neural turn models are unavailable, maintain a minimum silence window of 800 ms to 1000 ms. Sub-500 ms static silence timers consistently introduce unacceptable user cut-off rates on natural telephone speech.
- Implement Clean Buffer Flushing on Interruption: Ensure that barge-in events immediately cancel pending LLM generation tasks and clear audio output queues over WebRTC data channels to prevent delayed echo playback.
Sources
- Silero VAD Repository and Benchmarks
- PyAnnote.audio: Neural Building Blocks for Speaker Diarization
- LiveKit Turn Detector and Endpointing Architecture
- LiveKit Blog: Turn Detection for Voice Agents
- Cartesia: Voice Activity Detection for Real-Time Agents
- Picovoice: Voice Activity Detection Guide and Benchmarks
