Example of a Probability-Based Word Problem for LLMs
An example of a word problem that requires arithmetic and probability reasoning is: 'Three friends play a game where they flip a fair coin. Each friend flips the coin once. What is the probability that exactly one of them flips heads?' Solving this type of problem is challenging for LLMs. A common strategy for an LLM to begin its response is by signaling a structured, sequential thought process, for instance, by stating 'Let’s think step by step.'
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Example of a Probability-Based Word Problem for LLMs
Example of a Multi-Step Arithmetic Word Problem (Swimming Pool)
Example of a Mathematical Reasoning Word Problem (Jessica's Apps)
Example of a Multi-Step Arithmetic Word Problem (Tom's Marbles)
A large language model was given the following word problem: 'A bakery had 20 muffins. They sold 12 muffins and then baked 3 dozen more. How many muffins does the bakery have now?' The model produced this response: 'First, we start with 20 muffins. They sold 12, so 20 - 12 = 8. Then they baked 3 more, so 8 + 3 = 11. The final answer is 11.' Which statement best analyzes the primary reasoning failure in the model's response?
Chain-of-Thought (COT) Prompting
Example of a Multi-Step Arithmetic Word Problem (Jack's Apples)
Evaluating LLM Arithmetic Inference
A language model is tasked with solving arithmetic word problems. Below are common types of errors it might make when translating language into a sequence of mathematical operations. Match each error type with the scenario that best exemplifies it.
Learn After
Calculating the Probability of Exactly One Head in Three Coin Flips
An AI model is prompted with the following word problem: 'Three friends play a game where they flip a fair coin. Each friend flips the coin once. What is the probability that exactly one of them flips heads?'
Below are two different initial responses generated by the AI. Analyze the strategies and determine which response represents a more effective approach for solving this type of multi-step reasoning problem.
Response A: "The probability of one person getting heads is 1/2. Since there are three people, and we want exactly one to get heads, the probability is 1/3 multiplied by 1/2, which equals 1/6."
Response B: "Let’s think step by step. First, let's list all the possible outcomes for the three coin flips. Second, let's identify which of those outcomes have exactly one head. Third, we can calculate the probability by dividing the number of favorable outcomes by the total number of possible outcomes."
Evaluating AI Reasoning in a Probability Problem
LLM Reasoning on Probability Word Problems