Analyze the primary advantage and the primary disadvantage of adopting the 'select from predefined options' approach for this specific educational application. Frame your analysis in terms of answer quality and the model's evaluation.

Google

Multiple-choice question answering is a task format where a model is presented with a question and a set of predefined options, and it must select the correct answer from the given choices.

Multiple-Choice Question Answering

The MMLU (Massive Multitask Language Understanding) benchmark is a prominent example of how complex reasoning tasks are structured in a question-answering format. Each problem within this benchmark is presented as a multiple-choice question, requiring a Large Language Model to choose the single correct answer from a list of options.

MMLU Benchmark

A team of engineers is evaluating a new language model's reasoning capabilities. They use an assessment method where the model must choose the single correct answer from a set of provided options for each question. Which of the following represents a primary limitation of this evaluation method for gauging the model's genuine comprehension?

AI Tutor Design Strategy

Imagine you are designing a test to evaluate a new language model's ability to handle subtle linguistic nuances and common-sense reasoning. Your task is to create a single, original multiple-choice question that is easy for a human to answer but is likely to be challenging for a language model. Your response must include:
1. The question itself.
2. Four answer choices (one correct, three incorrect 'distractors').
3. A brief explanation of why the specific distractors you created are likely to mislead a language model, even if it has been trained on a vast amount of text.

Designing a Challenging Multiple-Choice Question for a Language Model

A grammaticality judgment prompt can be designed by first providing the sentence to be assessed, followed by a related question, and ending with clear multiple-choice options for the answer. For example, the sentence 'LLMs are powerful models but are expensive to build.' is presented at the beginning. This is followed by asking 'Does this sentence make sense grammatically?' and supplying an 'Answer Option' list that explicitly states the choices '- Yes' and '- No', along with the expected final response 'Yes'.

Learn Before

Related