Learn Before
Visual Inputs and Multimodal Processing - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Transferability of Language Prompting Techniques to Multimodal Inputs
Visual Input and Multimodal Processing - Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor
Example of GPT-4 Step-by-Step Visual Humor Explanation
An example demonstrating multimodal understanding and step-by-step reasoning involves presenting GPT-4 with a comic panel and the instruction: "Can you explain why this is funny. Think about it step-by-step." The comic depicts a statistical learning practitioner recommending complex, theoretical interventions for overfitting (such as structural risk minimization and loss function modification), contrasted with a neural network practitioner who simply suggests adding more layers. GPT-4 successfully identifies the humor arising from the contrast between theoretical mathematical complexity and the brute-force simplicity of stacking layers, noting that the caption "But unironically" reinforces the joke because adding layers is often effective in practice.
0
1
Contributors are:
Who are from:
Tags
Prep Sessions
Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Ch.2 Model Scaling and Capability Evaluation - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Visual Inputs and Multimodal Processing - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor
Ch.1 Foundation Model Capabilities and Benchmarking - Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor
Visual Input and Multimodal Processing - Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor
Related
Multimodal Input Processing in GPT-4
Transferability of Language Prompting Techniques to Multimodal Inputs
Example of GPT-4 Step-by-Step Visual Humor Explanation
GPT-4
Prompt for a Classification Task
What two core capabilities do multimodal models leverage across combined image and text tasks through the transfer of standard text-only prompting techniques?
A multimodal model fails to answer an analytical question about an intricate visual diagram accurately because it attempts to predict the final conclusion immediately without intermediate steps. Analyze how applying chain-of-thought (CoT) prompting resolves this failure, and explain how this language-derived method functions across combined image and text inputs.
Example of GPT-4 Step-by-Step Visual Humor Explanation
Match each multimodal task requirement to its corresponding prompt design strategy.
Standard prompting strategies developed for text-only language models—such as few-shot demonstrations and chain-of-thought prompting—are applied to multimodal systems. What is a key characteristic of how these techniques operate across image and text tasks?
Multimodal Input Processing in GPT-4
Transferability of Language Prompting Techniques to Multimodal Inputs
Example of GPT-4 Step-by-Step Visual Humor Explanation
Learn After
In GPT-4's step-by-step explanation of the comic panel, why does the caption "But unironically" reinforce the joke?
In the visual humor evaluation demonstration, GPT-4 was instructed to analyze the comic panel by thinking about it step-by-step.
Explain the core contrast between the two practitioners in the comic panel that GPT-4 identifies as the primary source of humor.
Name the two interventions recommended by the statistical learning practitioner to address overfitting in the comic panel analyzed by GPT-4.