Example of a Probability-Based Word Problem for LLMs

Example of a Multi-Step Arithmetic Word Problem (Swimming Pool)

Example of a Mathematical Reasoning Word Problem (Jessica's Apps)

Example of a Multi-Step Arithmetic Word Problem (Tom's Marbles)

A large language model was given the following word problem: 'A bakery had 20 muffins. They sold 12 muffins and then baked 3 dozen more. How many muffins does the bakery have now?' The model produced this response: 'First, we start with 20 muffins. They sold 12, so 20 - 12 = 8. Then they baked 3 more, so 8 + 3 = 11. The final answer is 11.' Which statement best analyzes the primary reasoning failure in the model's response?

Chain-of-Thought (COT) Prompting

Example of a Multi-Step Arithmetic Word Problem (Jack's Apples)

Evaluating LLM Arithmetic Inference

A language model is tasked with solving arithmetic word problems. Below are common types of errors it might make when translating language into a sequence of mathematical operations. Match each error type with the scenario that best exemplifies it.

Example of a Mathematical Reasoning Task for LLMs

Analyzing LLM Capabilities in Mathematical Problem-Solving

A developer is building an application that relies on a large language model to perform precise, multi-step arithmetic calculations. They notice the model occasionally produces small, difficult-to-predict errors in its final answers. Which of the following strategies would be the most reliable way to improve the system's mathematical accuracy?

Example of a Multi-Step Arithmetic Word Problem (Swimming Pool)

LLM Mathematical Reasoning Failure Analysis