Language models run on tokens, and the oldest number in the business is that text compresses to roughly 3 or 4 tokens per second of human speech. Audio does not cooperate. A single second of telephone-quality sound is 8,000 raw numbers. The standard sampling rate for modern speech models is 16,000 or 24,000 per second. If a model tried to predict raw audio samples the way it predicts words, a five-minute conversation would become millions of prediction steps, and the context window would be consumed before anyone finished a sentence.
The fix, developed over the past decade, is to stop treating audio as a special signal and turn it into a sequence of discrete units the same machinery already handles. Those units are audio tokens. They are the reason voice assistants can interrupt you, hold a conversation, and clone a voice from three seconds of tape.
The starting point: predicting samples one at a time
The first serious attempt was brute force. DeepMind's WaveNet, published in September 2016, was an autoregressive model of the raw waveform itself. Each audio sample was conditioned on all previous samples, generated one at a time at 16,000 samples per second, using stacks of dilated convolutions to give the model a long enough receptive field to capture the structure of speech. It worked: human listeners rated its English and Mandarin speech as significantly more natural than the best parametric and concatenative text-to-speech systems of the day. It was also miserable to run. Generating sound tens of thousands of times per second is computationally expensive, which is why the field looked for a compression step instead.
Whisper's detour: sound as a picture
Before anyone agreed on audio tokens, the most widely deployed speech model took a different route. OpenAI's Whisper, trained on 680,000 hours of audio-transcript pairs scraped from the web, does not read waveforms. It reads something closer to an image. Every input clip is resampled to 16 kHz and converted into an 80-channel log-mel spectrogram, computed with a 25 ms window and a 10 ms stride, then padded or cut to a fixed 30-second frame. A log-mel spectrogram is a time-frequency picture of sound: a sliding Fourier transform measures the energy in each frequency band, the frequency axis is warped to the mel scale, which stretches the ranges humans discriminate finely, and a logarithm compresses loudness. The transformer encoder then reads this two-dimensional surface and the decoder transcribes it.
This works extremely well for recognition. It is a one-way door for generation: a spectrogram is something you analyze, not something a language model naturally produces word by word. For talking, models needed discrete units.
The codec trick: compress sound into codebook entries
The modern answer comes from neural audio codecs. Google's SoundStream (2021) was the first fully learned, end-to-end codec of its kind. A convolutional encoder squashes a 24 kHz waveform into embeddings at a frame rate of 75 Hz, meaning one frame covers about 13.3 ms of sound. Each frame is then quantized into integers by a residual vector quantizer.

Residual vector quantization (RVQ) is the load-bearing idea. A single codebook that could represent 80 bits of information per frame would need 2 to the 80th power vectors, which the SoundStream paper notes is obviously unfeasible. Instead, quantizers run in series. The first picks the nearest codebook entry and leaves an error. The second quantizes that error, leaving a smaller error. Each further quantizer adds refinement, so a frame is represented by a short stack of small integers rather than one astronomically large one. Bitrate becomes a dial: SoundStream spans 3 to 18 kbps with one model, and in listening tests it beat the hand-engineered Opus codec at 12 kbps while using a quarter of the bits.
Meta's EnCodec (2022) open-sourced the recipe and pushed fidelity across bandwidths from 1.5 to 24 kbps, with a causal 24 kHz mono model whose initial latency is 13.3 ms. EnCodec became the default tokenizer for generative audio: MusicGen generates music by predicting EnCodec tokens, and VALL-E built speech synthesis on top of them.
Speech becomes a language modeling problem
Once audio is a token sequence, text-to-speech stops being signal regression and becomes the same next-token task a language model already does. Microsoft's VALL-E (January 2023) made that move explicit. It treats EnCodec's discrete codes as a target vocabulary and trains on LibriLight, 60,000 hours of English audiobook speech from roughly 7,000 speakers. Give it a transcript plus a 3-second enrollment clip of an unseen speaker, and it generates the clip's codec tokens conditioned on both, producing the voice, the emotion, and even the room acoustics of the prompt. It was the first demonstration that zero-shot voice cloning needs no speaker fine-tuning, only more tokens and a bigger dataset.
The cost is that audio tokens are still dense. At 75 Hz, one second of speech is 75 timesteps, and every timestep carries multiple codebook entries.
Slowing the clock down: Moshi and real-time conversation
Kyutai's Moshi (2024) attacked the density problem head-on, on the way to the first real-time full-duplex spoken language model. Its Mimi codec compresses 24 kHz audio down to 12.5 Hz, one frame every 80 ms, at about 1.1 kbps, and it does so in a fully streaming fashion. The frame rate is deliberate: it puts audio close to the 3 to 4 tokens per second of natural text, which keeps autoregressive generation compatible with live conversation. Mimi also borrows a lesson from recognition models. Its first codebook is distilled from WavLM, a self-supervised speech model, so the leading tokens carry semantic content and the trailing ones carry acoustics. A 7B-parameter temporal transformer then models two audio streams at once, Moshi's own voice and the user's voice, plus a time-aligned stream of text tokens it calls an inner monologue, which improves linguistic quality. The result is a system that listens while it speaks, with a theoretical latency of 160 ms and around 200 ms in practice, close to human response times and far below what a cascaded pipeline can achieve.

Meanwhile, the frontier labs moved toward the same destination from the other side. When OpenAI announced GPT-4o in May 2024, it described a single model trained end-to-end across text, vision and audio, replacing the ASR-then-LLM-then-TTS cascade that ChatGPT's voice mode had used. All inputs and outputs are processed by the same network, which is why it accepts interruptions mid-sentence.
What audio tokens lose
The conversion is lossy by construction. A codec at 1.5 to 6 kbps keeps what a decoder network needs to render plausible sound, not every measurement of the original waveform. Music and voice survive because the quantizers spend most of their bit budget on perceptually important structure. Exotic sounds, overlapping speakers, and precise timing details degrade first, and models trained on codec tokens inherit those blind spots. There is also a security side: the same VALL-E mechanism that clones a voice from three seconds of audio is the reason the paper became a reference point for voice-cloning risk.
Why this matters if you build with voice
The economics of voice AI in 2026 follow from these numbers. Frame rates decide token bills: a 75 Hz codec generates an order of magnitude more steps than a 12.5 Hz one for the same minute of audio. Latency floors are set by frame duration plus model time, which is why Moshi chose 80 ms frames. And any cascade that transcribes speech to text before reasoning throws away everything the transcript does not carry: tone, hesitation, the interruption. The labs betting on audio tokens are betting that the voice itself is the interface, not a wrapper around text.
A language model's whole world is sequences of integers. Audio tokens are what happens when sound finally agrees to join.
Sources
- Aaron van den Oord et al., "WaveNet: A Generative Model for Raw Audio," arXiv:1609.03499, September 2016. https://arxiv.org/abs/1609.03499
- DeepMind blog, "WaveNet: A generative model for raw audio," September 2016. https://deepmind.google/blog/wavenet-a-generative-model-for-raw-audio/
- Alec Radford et al., "Robust Speech Recognition via Large-Scale Weak Supervision" (Whisper), arXiv:2212.04356, December 2022. https://arxiv.org/abs/2212.04356
- Neil Zeghidour et al., "SoundStream: An End-to-End Neural Audio Codec," arXiv:2107.03312, July 2021. https://arxiv.org/abs/2107.03312
- Alexandre Défossez et al., "High Fidelity Neural Audio Compression" (EnCodec), arXiv:2210.13438, October 2022. https://arxiv.org/abs/2210.13438
- facebookresearch/encodec repository. https://github.com/facebookresearch/encodec
- Chengyi Wang et al., "Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers" (VALL-E), arXiv:2301.02111, January 2023. https://arxiv.org/abs/2301.02111
- Alexandre Défossez et al., "Moshi: a speech-text foundation model for real-time dialogue," arXiv:2410.00037, September 2024. https://arxiv.org/abs/2410.00037
- Kyutai Labs, moshi repository (Mimi codec details). https://github.com/kyutai-labs/moshi
- OpenAI, "Hello GPT-4o," May 2024. https://openai.com/index/hello-gpt-4o/
