Vector Quantization

Vector Quantization and VQ-VAEs: How Discrete Codebooks, Straight-Through Estimators, and Commitment Losses Power Multimodal Tokenization

Autoregressive sequence models excel at discrete token prediction. In natural language processing, words and subwords map onto categorical vocabularies where token identity is exact and cross-entropy loss provides direct likelihood optimization. Continuous multi-dimensional signals—such as images, video frames, raw audio waveforms, and robotic sensorimotor trajectories—present a fundamental mismatch for standard transformer architectures. Historically, variational autoencoders (VAEs) bridged ra