GSM8K Benchmark

Insufficiency of Simple Demonstrations for LLM Reasoning Tasks

A user gives a language model the following prompt: 'I have a box that contains a red ball and a blue ball. I take the red ball out and put it on the table. What is left in the box?' The model responds: 'The box contains a red ball and a blue ball.' Which of the following best analyzes the likely cause of the model's incorrect answer?

Commonsense Reasoning as a Challenging Task for LLMs

In-Context Learning (ICL)

The Challenge of Multi-Step Logical Inference for LLMs in Arithmetic Reasoning

Language Model Scheduling Error Analysis

Predicting LLM Reasoning Flaws

A developer provides a large language model with the following input text:

`Translate the following user requests into a structured command format.

Example 1: Request: "Set a timer for 10 minutes" Command: {"action": "set_timer", "duration_minutes": 10}

Example 2: Request: "What's the weather in London?" Command: {"action": "get_weather", "location": "London"}

Now, process this request: Request: "Play the new song by The Weeknd"`

The model correctly outputs: {"action": "play_music", "artist": "The Weeknd"}.

Which statement best analyzes the primary mechanism the model used to generate the correct command for the new request?

In-Context Learning (ICL)

True or False: When a large language model successfully performs a novel task after being shown examples within a single prompt, this indicates that the model has undergone a permanent update to its internal weights, effectively 'training' it on the new task for all future interactions.

Analyzing Model Behavior Across Sessions

Effect of 'Thinking' Prompts on LLM Performance

Chain-of-Thought (COT) Prompting

Multi-Round Interaction to Guide LLM Reasoning

Example of a Prompt for a Direct Mathematical Calculation

Example of a Prompt for Calculating the Average of 1, 3, 5, and 7

Example of a Prompt for Calculating the Mean Square

Improving LLM Reasoning with Step-by-Step Demonstrations

In-Context Learning (ICL)

A user is trying to get a Large Language Model (LLM) to solve a complex word problem that involves multiple calculations. Their initial prompt, 'What is the answer to this problem? [Problem text]', results in a quick but incorrect numerical answer. The user then revises the prompt to: 'First, break down the problem into the necessary steps. Then, solve each step, showing your work. Finally, state the final answer. [Problem text]'. This revised prompt leads to a correct solution. Which principle of interacting with LLMs does this scenario best illustrate?

Evaluating Prompt Strategies for a Logic Puzzle

Prompting for a Reasoning Process to Mitigate Errors in Complex Tasks

Example of a Prompt for Calculating the Average of 2, 4, and 9

Improving LLM Problem-Solving by Demonstrating Reasoning Steps

The Mechanism of Reasoning Prompts

Example of a Prompt with Detailed Reasoning Steps