Evaluating AI Reasoning in a Probability Problem
An AI model is given a probability word problem and produces the response below. Although the final answer is correct, evaluate the reliability of the model's reasoning process. Explain why this process might be considered weak or unreliable for solving similar, more complex problems.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Calculating the Probability of Exactly One Head in Three Coin Flips
An AI model is prompted with the following word problem: 'Three friends play a game where they flip a fair coin. Each friend flips the coin once. What is the probability that exactly one of them flips heads?'
Below are two different initial responses generated by the AI. Analyze the strategies and determine which response represents a more effective approach for solving this type of multi-step reasoning problem.
Response A: "The probability of one person getting heads is 1/2. Since there are three people, and we want exactly one to get heads, the probability is 1/3 multiplied by 1/2, which equals 1/6."
Response B: "Let’s think step by step. First, let's list all the possible outcomes for the three coin flips. Second, let's identify which of those outcomes have exactly one head. Third, we can calculate the probability by dividing the number of favorable outcomes by the total number of possible outcomes."
Evaluating AI Reasoning in a Probability Problem
LLM Reasoning on Probability Word Problems