Vision-Language Model Architectures: How Vision Encoders, Token Projectors, and Dynamic Resolution Bridge Modalities
Modern large language models operate exclusively on discrete sequences of token embeddings. Integrating visual perception into these systems requires converting continuous, two-dimensional spatial arrays into linear sequences of embedding vectors that match the dimensionality and distribution of the language model's hidden states. Vision-language models (VLMs) like LLaVA, Flamingo, Qwen2-VL, and InternVL accomplish this conversion through a three-stage pipeline: a vision encoder, a cross-modal
1 min