University of Michigan - Ann Arbor

GPT-4 accepts inputs composed of arbitrarily interlaced text and images, enabling users to specify vision and language tasks in a manner parallel to text-only settings. Across a diverse range of visual domains—including documents containing mixed text and photographs, diagrams, and screenshots—the model generates text outputs while demonstrating capabilities comparable to its performance on purely text-based inputs.

Multimodal Input Processing in GPT-4

When evaluating multimodal inputs across diverse visual domains, what type of output does GPT-4 produce?

How does GPT-4 allow users to combine text and images within a single input?

Standard test-time prompting methods developed for text-only language models—including few-shot demonstration prompting and chain-of-thought (CoT) prompting—remain directly applicable and effective when working with multimodal inputs. This enables multimodal models to leverage structured reasoning elicitation and in-context examples across combined image and text tasks.

Learn Before

Related