GSM8K Benchmark

Insufficiency of Simple Demonstrations for LLM Reasoning Tasks

A user gives a language model the following prompt: 'I have a box that contains a red ball and a blue ball. I take the red ball out and put it on the table. What is left in the box?' The model responds: 'The box contains a red ball and a blue ball.' Which of the following best analyzes the likely cause of the model's incorrect answer?

Commonsense Reasoning as a Challenging Task for LLMs

The Challenge of Multi-Step Logical Inference for LLMs in Arithmetic Reasoning

Language Model Scheduling Error Analysis

Predicting LLM Reasoning Flaws

In-Context Learning

A developer provides a large language model with the following input text:

`Translate the following user requests into a structured command format.

Example 1: Request: "Set a timer for 10 minutes" Command: {"action": "set_timer", "duration_minutes": 10}

Example 2: Request: "What's the weather in London?" Command: {"action": "get_weather", "location": "London"}

Now, process this request: Request: "Play the new song by The Weeknd"`

The model correctly outputs: {"action": "play_music", "artist": "The Weeknd"}.

Which statement best analyzes the primary mechanism the model used to generate the correct command for the new request?

True or False: When a large language model successfully performs a novel task after being shown examples within a single prompt, this indicates that the model has undergone a permanent update to its internal weights, effectively 'training' it on the new task for all future interactions.

Analyzing Model Behavior Across Sessions

In-Context Learning

Effect of 'Thinking' Prompts on LLM Performance

Chain-of-Thought (COT) Prompting

Multi-Round Interaction to Guide LLM Reasoning

Example of a Prompt for a Direct Mathematical Calculation

Example of a Prompt for Calculating the Average of 1, 3, 5, and 7

Example of a Prompt for Calculating the Mean Square

Improving LLM Reasoning with Step-by-Step Demonstrations

A user is trying to get a Large Language Model (LLM) to solve a complex word problem that involves multiple calculations. Their initial prompt, 'What is the answer to this problem? [Problem text]', results in a quick but incorrect numerical answer. The user then revises the prompt to: 'First, break down the problem into the necessary steps. Then, solve each step, showing your work. Finally, state the final answer. [Problem text]'. This revised prompt leads to a correct solution. Which principle of interacting with LLMs does this scenario best illustrate?

Evaluating Prompt Strategies for a Logic Puzzle

Prompting for a Reasoning Process to Mitigate Errors in Complex Tasks

Example of a Prompt for Calculating the Average of 2, 4, and 9

Improving LLM Problem-Solving by Demonstrating Reasoning Steps

The Mechanism of Reasoning Prompts

Example of a Prompt with Detailed Reasoning Steps

In-Context Learning

The creators of the large-scale, unsupervised language model introduced in 2019 initially withheld the full version from the public, citing concerns about potential misuse. Which statement best evaluates the significance of this 'staged release' strategy for the field of artificial intelligence?

Analysis of Model Scaling Impact

Evaluating Model Capabilities in a Research Scenario

In-Context Learning

In-Context Learning (ICL)

Megatron-Turing NLG

Gopher

CLIP (Contrastive Language-Image Pre-training)