Learn Before
Example of an MMLU Question (Abstract Algebra)
An example of a question from the MMLU benchmark, taken from the abstract algebra subject, is presented as follows:
is a group with for all , in . The inverse of is (A) (B) (C) (D) D
This illustrates the multiple-choice format used in the benchmark to test a model's reasoning capabilities on specialized topics.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Foundations of Large Language Models Course
Related
Example of an MMLU Question (Abstract Algebra)
Match each MMLU benchmark component to its role or definition described in the course content.
Explain why this researcher's proposed protocol conflicts with the format mandated by the MMLU benchmark, and identify what the model is required to produce instead.
According to the course content, which broad category of tasks does the MMLU benchmark organize into a question-answering format?
What full name does the acronym "MMLU" represent?
Multilingual Benchmark Translation for LLM Evaluation
Learn After
Analysis of a Reasoning-Based Benchmark Question
Consider the following problem, which is an example from a benchmark designed to test the reasoning capabilities of advanced AI models:
A mathematical structure is defined by the set of all integers and an operation
*such that for any two integersaandb,a * b = a + b + 1. This structure forms a group. What is the inverse of an elementawithin this group?Analysis of a Benchmark Reasoning Problem