Multimodal AI is AI that can work with more than one kind of data at once: text, images, audio and video. When you photograph a broken dishwasher part and ask a chatbot what it is, or talk to a voice assistant that hears your tone, you are using a multimodal model. This explainer covers what the term means, how these models are built, the three main designs, and where they still fail.
What "modality" means
A modality is a type of input or output. Text is one. Images, audio, video and sensor data are others. A text-only model handles one: text in, text out. A multimodal model accepts or produces at least two modalities.
The words get used loosely, so it helps to separate two things:
- Multimodal input: the model can read images, audio or video along with your text.
- Multimodal output: the model can also produce something other than text, such as speech or an image.
OpenAI's 2024 GPT-4o announcement is a clear example of both. OpenAI says GPT-4o "accepts as input any combination of text, audio, image, and video and generates any combination of text, audio, and image outputs." Google described its Gemini family as "a new family of multimodal models" with capabilities "across image, audio, video, and text understanding."
Why it matters for everyday use
The world is not only text. Receipts, charts, handwritten notes, screenshots, voice memos and videos all carry information that a text-only model cannot see. Multimodal models let you skip the step of describing the thing to the AI. You show it instead.
There is also a quality reason. OpenAI's GPT-4o post explains what was lost in the older approach. Before GPT-4o, ChatGPT's Voice Mode was "a pipeline of three separate models": one transcribed audio to text, GPT-3.5 or GPT-4 handled the text, and a third model turned the reply back into audio. OpenAI says this meant the main model "loses a lot of information": it "can't directly observe tone, multiple speakers, or background noises, and it can't output laughter, singing, or express emotion." OpenAI says that with GPT-4o it "trained a single new model end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network."
Speed changed too. OpenAI reports that the old Voice Mode had average latencies of 2.8 seconds with GPT-3.5 and 5.4 seconds with GPT-4. It says GPT-4o "can respond to audio inputs in as little as 232 milliseconds, with an average of 320 milliseconds."
How a model turns pictures and sound into something it can read
A language model works on tokens: small chunks of text turned into lists of numbers (vectors). If you want the same model to handle an image, you need a way to turn that image into vectors the model can process alongside the words. Our tokenizer tax explainer covers how text tokens work and why their count drives cost.
For images, one method comes from the 2020 Google paper An Image is Worth 16x16 Words. It showed that "a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks." The idea: cut the image into small square patches, turn each patch into a vector, and feed the sequence of patches to a transformer the same way you would feed a sentence of words. That model is called a Vision Transformer, or ViT.
For audio, a model such as OpenAI's Whisper learns to map speech to text. The Whisper paper reports training on 680,000 hours of "multilingual and multitask supervision," and says the resulting models "generalize well to standard benchmarks" without fine-tuning.
Teaching a model that a picture and a caption mean the same thing
Turning an image into vectors is not enough. The model also has to learn that the vector for a photo of a dog sits near the vector for the word "dog." One way to teach this comes from OpenAI's 2021 CLIP paper.
CLIP was trained on a simple task: "predicting which caption goes with which image." The researchers used "a dataset of 400 million (image, text) pairs collected from the internet." The result was an image encoder that understands visual concepts described in plain language. The paper says that after training, "natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer." In one test, CLIP matched the accuracy of the original ResNet-50 on ImageNet "without needing to use any of the 1.28 million training examples it was trained on."
CLIP itself does not chat. But a 2024 study covered below found that the visual part of recent multimodal chatbots typically rests on CLIP pre-training. Our technical CLIP explainer covers how its training works in detail.
The three main designs
This explainer groups multimodal systems into three designs. Each makes a different trade between cost, simplicity and how much the model can perceive.
1. The pipeline
Separate specialist models are chained together, and they pass text between them. Speech goes to a transcription model, the text goes to a language model, and the reply goes to a text-to-speech model. ChatGPT's pre-GPT-4o Voice Mode, as OpenAI describes it, worked this way.
- Upside: easy to build from parts that already exist.
- Downside: each handoff loses information and adds delay. The language model only ever sees a transcript, so tone and background sound never reach it.
2. A vision encoder attached to a language model
A pretrained image encoder is connected to a pretrained language model, and the pair is trained to work together. Our technical explainer on vision-language model architectures walks through this design in models such as LLaVA, Qwen2-VL and InternVL.
DeepMind's 2022 Flamingo paper was an early example. The authors describe architectural changes to "bridge powerful pretrained vision-only and language-only models" and to "handle sequences of arbitrarily interleaved visual and textual data." Flamingo could be trained on web pages that mix text and images, which the authors say was key to its ability to learn new tasks from a few examples in the prompt.
The 2023 LLaVA paper is another example. LLaVA "connects a vision encoder and LLM for general-purpose visual and language understanding." The team used text-only GPT-4 to generate image-based instruction-following data and then tuned the combined model on it. They released the data, model and code.
- Upside: reuses strong pretrained models instead of training everything from scratch.
- Downside: the language model only sees what the vision encoder passes it. If the encoder misses a detail, the chatbot cannot recover it (see limits below).
3. Early fusion: one model, one stream
In this design, images and text are both turned into tokens and placed in a single sequence that one model is trained on from the start. Meta's 2024 Chameleon paper describes "early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence." The authors say this is "a significant step forward in a unified modeling of full multimodal documents," and they outline "a stable training approach from inception."
OpenAI's description of GPT-4o, with all inputs and outputs "processed by the same neural network," points the same way, although OpenAI's post does not give architecture details.
- Upside: the model can reason across modalities directly and can produce mixed output, such as text with images, in one pass.
- Downside: training is harder to keep stable. The Chameleon authors write that it "was challenging to maintain stable training when scaling the Chameleon models above 8B parameters and 1T tokens," and they changed the transformer architecture to fix it.
What multimodal models are good at today
The papers above test multimodal models on these jobs:
- Describing and answering questions about images. Flamingo and Gemini were evaluated on visual question answering and captioning. LLaVA was tested on image-based chat and on Science QA.
- Reading text in images. CLIP was benchmarked on tasks that include OCR (reading printed text in pictures).
- Speech recognition across languages. Whisper's results come from training on 680,000 hours of multilingual audio.
- Real-time voice conversation. OpenAI says GPT-4o's 320 millisecond average audio response time is "similar to human response time in a conversation."
For a practical example, try photographing a chart and asking a chatbot to explain the trend, or uploading a screenshot of an error message and asking what it means. These are the everyday cases where multimodal input saves you typing.
Where they still fail
Multimodal models make confident mistakes, just as text models do. Our hallucinations explainer covers the text side. The visual side has its own weak spots.
A 2024 study titled Eyes Wide Shut? looked at why. The authors note that the visual component of recent multimodal LLMs "typically depends only on" CLIP pre-training. They found "CLIP-blind pairs": "images that CLIP perceives as similar despite their clear visual differences." From these, they built the MMVP benchmark, which tests nine basic visual patterns. They report that state-of-the-art systems, "including GPT-4V, struggle with straightforward questions" on it, "often providing incorrect answers and hallucinated explanations." The authors conclude that "visual representation learning remains an open challenge."
In practice, that means:
- Double-check counts, small details and positions. Questions like "how many" or "which way is it facing" are the kind of basic visual patterns the MMVP study tests.
- Do not treat a description as proof. A model can describe an image fluently and still get a key detail wrong.
- Give it clear inputs. A sharp, well-lit photo or a clean screenshot gives the vision encoder more to work with than a blurry one.
OpenAI also said at the GPT-4o launch that audio brings "a variety of novel risks," and that at launch audio outputs would be "limited to a selection of preset voices."
Multimodal AI and other terms you will see
A few related terms often come up next to this one:
- Vision-language model (VLM): a model that handles images and text. Flamingo's authors call their models "Visual Language Models (VLM)."
- Large multimodal model (LMM): a general term for a large model that handles several modalities. The LLaVA paper calls LLaVA "an end-to-end trained large multimodal model."
- Omni model: a label used in some product names for models that take in and produce many modalities. The "o" in GPT-4o stands for "omni," per OpenAI.
- Retrieval-augmented generation (RAG): a separate idea about feeding a model documents at answer time. Our RAG explainer covers the basics.
The short version
Multimodal AI means one system that can take in, and sometimes produce, more than text. This explainer covered three designs: chaining specialist models together, attaching a vision or audio encoder to a language model, and training one model on mixed tokens from the start. OpenAI's own account of GPT-4o explains why it moved away from the first design: each handoff between separate models loses information. The weak spot that remains is perception itself: models can still misread simple visual details while sounding sure of themselves, so check anything that matters.
Sources
- OpenAI: Hello GPT-4o
- Gemini: A Family of Highly Capable Multimodal Models (Google, arXiv)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Google, arXiv)
- Robust Speech Recognition via Large-Scale Weak Supervision, the Whisper paper (OpenAI, arXiv)
- Learning Transferable Visual Models From Natural Language Supervision, the CLIP paper (OpenAI, arXiv)
- Flamingo: a Visual Language Model for Few-Shot Learning (DeepMind, arXiv)
- Visual Instruction Tuning, the LLaVA paper (arXiv)
- Chameleon: Mixed-Modal Early-Fusion Foundation Models (Meta, arXiv)
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs (arXiv)
