Evaluating Prompting Strategies for High-Stakes Tasks
A development team is building a system to answer complex legal questions using a large language model. They are debating between two prompting strategies:
- A detailed, multi-step reasoning prompt that guides the model through a specific logical sequence.
- A simpler, direct prompt that asks for the final answer without specifying the reasoning steps.
Evaluate the potential risks and benefits of choosing the detailed, multi-step reasoning approach over the simpler, direct approach for this specific application. In your evaluation, analyze at least two distinct challenges inherent to the multi-step method.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Difficulty of Creating Few-Shot CoT Demonstrations
Lack of Standardized Problem Decomposition in CoT
Error Propagation in CoT Reasoning Steps
Diagnosing a Reasoning Failure in a Multi-Step Prompt
A team is using a large language model to perform complex multi-step financial analysis. They provide the model with several examples of how to break down a problem and arrive at a conclusion. However, they notice that the model's performance is inconsistent. Prompts created by senior analysts, who use a methodical approach to breaking down the problem, yield reliable results. In contrast, prompts created by junior analysts, who each use their own ad-hoc approach, often lead the model to make logical errors early in its reasoning process. Which of the following interventions would most directly address the root cause of this inconsistency?
Evaluating Prompting Strategies for High-Stakes Tasks