Learn Before
Designing a Challenging Multiple-Choice Question for a Language Model
Imagine you are designing a test to evaluate a new language model's ability to handle subtle linguistic nuances and common-sense reasoning. Your task is to create a single, original multiple-choice question that is easy for a human to answer but is likely to be challenging for a language model. Your response must include:
- The question itself.
- Four answer choices (one correct, three incorrect 'distractors').
- A brief explanation of why the specific distractors you created are likely to mislead a language model, even if it has been trained on a vast amount of text.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Creation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
MMLU Benchmark
A team of engineers is evaluating a new language model's reasoning capabilities. They use an assessment method where the model must choose the single correct answer from a set of provided options for each question. Which of the following represents a primary limitation of this evaluation method for gauging the model's genuine comprehension?
AI Tutor Design Strategy
Designing a Challenging Multiple-Choice Question for a Language Model
Example of a Sentence-First Prompt for Grammaticality Judgment with Answer Options