Open-Source Text-to-Speech (TTS) Serving in Production: Comparing Kokoro-82M, F5-TTS, StyleTTS 2, and Fish Speech

Deploying open-source text-to-speech (TTS) in production has shifted rapidly over the past year. Where teams previously defaulted to managed cloud APIs like ElevenLabs or OpenAI Audio due to the complexity and computational weight of self-hosted acoustic models, modern open-weight architectures offer distinct trade-offs across latency, zero-shot cloning fidelity, and inference cost.

Selecting a self-hosted TTS engine requires navigating four fundamentally different architectural paradigms: lightweight non-diffusion acoustic decoders, non-autoregressive conditional flow-matching diffusion transformers, style diffusion with adversarial vocoding, and dual-autoregressive discrete speech codec language models.

This analysis compares four leading open-source architectures in production: Kokoro-82M, F5-TTS, StyleTTS 2, and Fish Speech (S1/S2).

Architectural Paradigms

Kokoro-82M: Lightweight ISTFTNet Decoder

Developed by hexgrad, Kokoro-82M strips away heavy diffusion loops and auto-regressive token sampling in favor of an 82-million-parameter streamlined acoustic network derived from StyleTTS 2 and ISTFTNet. It uses fixed phoneme-to-acoustic mappings paired with pre-extracted style vectors and an inverse short-time Fourier transform (iSTFT) neural decoder.

Because it operates as a single deterministic forward pass without iterative ODE/SDE solving, Kokoro-82M achieves extreme inference speed on commodity CPUs and minimal GPU memory consumption. However, it lacks native zero-shot voice cloning from arbitrary audio prompts, relying instead on a curated set of preset voice profiles.

F5-TTS: Non-Autoregressive Flow Matching

Introduced by Chen et al. from Shanghai Jiao Tong University, F5-TTS treats speech synthesis as conditional integration along an optimal-transport probability path between an isotropic Gaussian prior and clean mel-spectrogram distributions.

The architecture pairs a 335.8-million-parameter Diffusion Transformer (DiT) backbone with a ConvNeXt V2 text representation module and a Vocos neural vocoder. F5-TTS utilizes filler-padding rather than complex duration prediction networks, allowing text and audio prompts to be concatenated directly. During inference, an Ordinary Differential Equation (ODE) numerical solver (typically Euler with 32 steps) integrates the learned vector field, enabling robust zero-shot voice cloning and cross-lingual synthesis.

StyleTTS 2: Style Diffusion and Adversarial Modeling

Published by Li et al. from Columbia University, StyleTTS 2 models speech synthesis using style diffusion conditioned on large speech language models (SLMs). It utilizes a non-autoregressive generator trained with generative adversarial networks (GANs), differentiable duration modeling, and path integral-based alignment via Monotonic Alignment Search (MAS).

StyleTTS 2 decouples prosodic style generation from acoustic synthesis. The diffusion model samples a style latent representation from text embeddings, which conditions an adversarial generator to synthesize time-domain audio. This yields high prosodic naturalness and stability across long documents, though exact zero-shot speaker cloning requires clean reference style extraction.

Fish Speech (S1/S2): Dual-Autoregressive Codec LLM

Developed by Fish Audio, the Fish Speech architecture models speech as discrete acoustic tokens using a dual-autoregressive Transformer architecture. The system tokenizes continuous audio into discrete multi-codebook representations via high-compression neural audio codecs (such as Fish-Codec/Dual-AR).

Operating identically to modern decoder-only language models, Fish Speech generates semantic and acoustic tokens sequentially from text and conditioning audio prompts. This provides fine-grained controllable prosody, non-verbal vocalizations (including laughter, sighs, and emotional inflection tags), and strong zero-shot voice cloning. However, auto-regressive generation introduces quadratic sequence scaling and higher Time-to-First-Audio latency.

Latency, Real-Time Factor, and Compute Footprint

Production serving performance is primarily evaluated using two metrics:

• Real-Time Factor (RTF): Total generation time divided by generated audio duration (lower values represent faster execution).
• Time-to-First-Audio (TTFA): Latency from request receipt to the emission of the first playable audio chunk.

Voice Cloning Fidelity vs. Word Error Rate Trade-offs

A core engineering trade-off in neural speech generation lies between speaker similarity (SIM) and intelligibility/robustness.

Zero-Shot Robustness in Flow Matching

F5-TTS achieves high speaker similarity scores in zero-shot cloning because the ODE trajectory integrates directly across the prompt audio's mel-spectrogram conditioning frames. However, because F5-TTS relies on text-guided filler padding rather than explicit phoneme duration aligners, missing or malformed punctuation in reference transcripts can induce word repetitions or deletions during the 32-step ODE sampling process.

Hallucination Control in Autoregressive Codec Models

Fish Speech treats speech as discrete token prediction, allowing it to inherit in-context learning capabilities. When provided with a 3-second reference clip, it accurately replicates vocal timbre, room acoustics, and speaking cadence.

However, like text LLMs, autoregressive acoustic models are susceptible to token-level hallucinations, phonetic drift, and infinite repetition loops on out-of-distribution phoneme clusters. Mitigating these failures requires KV-cache constraint masks, temperature tuning, and repetition penalties at serving time.

Deterministic Stability in Fixed-Acoustic Decoders

Kokoro-82M sacrifices open-ended voice cloning to guarantee high textual intelligibility and low error rates. By using deterministic duration extraction and phonemization pipelines (such as espeak-ng), it bypasses both ODE drift and autoregressive repetition failures. For long-form reading, audiobooks, and IVR pipelines where voice consistency is valued over dynamic voice cloning, deterministic decoders eliminate synthesis hallucinations.

Serving Stack Architecture and Quantization Limits

Deploying these architectures at scale requires tailored runtime runtimes and careful quantization strategies:

Flow Matching and ODE Quantization Degradation

In flow-matching architectures like F5-TTS, the 32-step iterative ODE solver evaluates the 22-layer DiT backbone twice per step to compute Classifier-Free Guidance (CFG). As a result, every model weight matrix is traversed up to 1,408 times per inference pass.

Because numerical errors compound multiplicatively across iterative solver steps:

• Standard 8-bit post-training quantization (such as GGUF Q8_0 or INT8 GEMM) introduces error margins that degrade acoustic trajectories, resulting in distorted audio artifacts.
• Production deployments of F5-TTS must maintain FP16 or BF16 execution across the DiT backbone, using TensorRT or TorchScript compilation to maximize Tensor Core throughput rather than aggressive sub-8-bit quantization.

ONNX Runtime and INT8 Acceleration in Kokoro

Conversely, Kokoro-82M does not execute iterative sampling loops. Its feed-forward ISTFTNet decoder can be exported to ONNX Runtime and quantized to INT8 with minimal perceptual degradation. This enables microsecond-level TTFA and allows full serving pipelines to run on edge instances, container sidecars, or lightweight CPU servers without dedicated GPU hardware.

WebRTC and Chunked Streaming Topologies

For interactive conversational voice agents (built with frameworks like LiveKit or Pipecat), streaming audio chunk delivery is required.

• Autoregressive Models (Fish Speech): Emit audio chunks as soon as discrete codec frames are decoded by the vocoder, providing low TTFA (~100 ms to 150 ms) on warm GPU instances despite higher total RTF.
• Non-Autoregressive Models (F5-TTS, Kokoro): Synthesize entire sentences or clauses in single batched sweeps. To achieve low latency in conversational loops, text must be segmented at punctuation boundaries (clauses of 5 to 15 words) before dispatching parallel inference requests.

Production Sizing and Economic Decision Matrix

When sizing infrastructure for production workloads, engineering teams should evaluate their primary operational constraints:

1. High-Throughput / Fixed Voices (Audiobooks, Notifications, Standard Agents):

• Recommendation: Deploy Kokoro-82M on CPU clusters or low-cost GPU instances (NVIDIA T4 / L4).
• Economics: At an RTF of 0.04 on an L4 instance ($0.38/hour), a single GPU can process over 25 concurrent audio streams, bringing generation costs below $0.01 per million characters.

1. High-Fidelity Zero-Shot Voice Cloning (Content Personalization, Dubbing):

• Recommendation: Deploy F5-TTS on NVIDIA A10G, L40S, or A100 GPUs using FP16 TensorRT pipelines.
• Economics: Generates high-similarity cloned voices from short reference samples with neutral prosody, requiring ~3 GB VRAM per worker.

1. Expressive Conversational Voice Agents with Emotion Tags:

• Recommendation: Deploy Fish Speech on high-memory GPU nodes (A100 / H100) paired with an optimized KV-cache serving engine.
• Economics: Delivers natural conversational cues and laughter at the cost of higher compute footprint and stricter monitoring for token repetition anomalies.

Sources

• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training
• Fish Audio S2 Technical Report
• Kokoro-82M Open Weights Repository
• T5Gemma-TTS Technical Report and Voice Cloning Benchmarks