Multimodal Models: Alignment, Fusion, and Visual Questions
Multimodal models combine information from text, images, audio, or video, but “supports images” does not specify a task. Finding photos from a sentence needs image–text similarity. Answering a question about a screenshot needs visual evidence connected to language generation. Producing an image is a different output objective. Start with inputs, outputs, and details that must survive processing.
This page uses images and text to explain common mechanisms. Audio involves waveforms or acoustic features; video adds temporal order. Neither reduces simply to more pictures. For image generation, see diffusion models.
Turning images into sequences
A basic Vision Transformer (ViT) divides an image into patches, flattens their pixels, projects each into a vector, and processes the resulting sequence. The image need not first become a textual description.
In a simplified example, a 224×224 image with 16×16 patches gives patches. Each RGB patch contains values before projection to width . Ignoring a classification token, visual states have shape [batch,196,d_v]. The pixel count 768 does not require a hidden width of 768.
Position information preserves spatial order. Preprocessing determines resizing, cropping, and normalization. Real systems may use high-resolution tiles, dynamic resolution, or compression, so 196 tokens per image is not a universal billing or memory rule. Visual patches are also generally continuous vectors; they need not correspond to integer IDs in the text vocabulary.
Alignment makes modalities comparable
CLIP produces vectors with separate image and text encoders and uses paired contrastive training to increase matching similarity. Neither side needs to generate a caption. This suits image–text retrieval and comparisons against text candidates.
Imagine three images with three paired descriptions. Their normalized vectors each have shape [3,d]. Matrix multiplication produces a [3,3] similarity table with correct pairs on the diagonal. Training favors the correct entry in each row and column. A nominal negative description that also fits an image creates noisy supervision.
At inference, compare an image with “red car,” “blue car,” and “bicycle.” Applying softmax makes probabilities conditional on this candidate set: removing an option or adding a similar description changes the distribution. It is not a universal open-world probability that a red car is present. Evaluate thresholds on the actual task distribution.
Open full-size imageRead the left panel as training: the encoders produce image and text vectors, and the highlighted diagonal contains the paired examples whose similarity should increase. The right panels show inference: turn class names into text descriptions, encode them, then compare one image against those vectors. The selected dog description is a candidate label, not an independently generated caption.
Fusion connects evidence to generation
Answering “What is the second error line in this screenshot?” usually needs more than whole-image similarity. The original LLaVA design connects a pretrained vision encoder to a language model, projects visual features to the language representation width, and trains answers using visual instructions.
A simplified connector multiplies [batch,N,d_v] by [d_v,d_l] to obtain [batch,N,d_l]. Matching width enables computation; training teaches the model to use those values. A random connector with correct dimensions does not create visual understanding. Other designs use resamplers or cross-attention, so one implementation does not define all multimodal models.
Training can freeze some components, train the connector, and then adapt to tasks; other procedures update more parameters. These choices affect cost and capability. Adding a projection is different from training cross-modal alignment. Text output still depends on decoding and stopping rules.
Information can disappear before answering
Resizing can erase small text; cropping can remove edges; compressing many visual positions can lose fine relations. A longer, more fluent answer cannot recover evidence absent from the processed input.
Recognizing a broad scene also differs from reliable counting, coordinate extraction, or left–right reasoning. A model may fill in an object from learned language expectations when that object often accompanies a scene. Reading several cells correctly does not establish correct row and column relationships in a table screenshot.
Video adds temporal sampling: a frame every few seconds may miss a brief event or obscure order. Audio transcription may discard tone, overlapping speakers, and non-speech sounds. Tasks needing this information should evaluate the original-modality pipeline, not only answers to a transcript.
Test dependence on evidence
Build a small public, controlled set in which a simple image changes only in color, count, left–right position, or written text. Keep the question fixed and check whether the answer follows the change. Then hide the decisive region and check whether the model remains equally certain. This is an experiment design, not a reported benchmark.
Locate failures in preprocessing, representation, the connection to the language model, or decoding. For OCR, record character errors and layout relations; for counting, absolute error; for questions, correctness and behavior without evidence. Compare models using the same resolution, crops, questions, and output budgets, while separately recording latency and visual-token counts.
For image–text search, connect alignment models to the retrieval pipeline. For screenshot questions, test detail visibility and evidence dependence. For multimodal agents, separate action verification from visual judgment: describing a button does not establish reliable localization or confirmation of the state after a click.