Diagnosing a Reasoning Failure in a Multi-Step Prompt
An engineer is testing a language model's ability to follow a sequence of instructions. The model is given the prompt and produces the reasoning trace below, which leads to an incorrect final answer. Based on the provided trace, identify the primary practical limitation of this prompting technique that is being demonstrated and explain why this example is a clear illustration of that limitation.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Difficulty of Creating Few-Shot CoT Demonstrations
Lack of Standardized Problem Decomposition in CoT
Error Propagation in CoT Reasoning Steps
Diagnosing a Reasoning Failure in a Multi-Step Prompt
A team is using a large language model to perform complex multi-step financial analysis. They provide the model with several examples of how to break down a problem and arrive at a conclusion. However, they notice that the model's performance is inconsistent. Prompts created by senior analysts, who use a methodical approach to breaking down the problem, yield reliable results. In contrast, prompts created by junior analysts, who each use their own ad-hoc approach, often lead the model to make logical errors early in its reasoning process. Which of the following interventions would most directly address the root cause of this inconsistency?
Evaluating Prompting Strategies for High-Stakes Tasks