Learn Before
Critique of a Prompt Finding Method
A colleague proposes a method for finding an effective prompt: generate a large, fixed set of 100 diverse prompts, evaluate all of them on a dataset, and select the single best-performing one. From the perspective of a systematic search process, what is the main conceptual limitation of this approach, and what fundamental characteristic should a more effective search strategy incorporate?
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Iterative LLM-Based Prompt Search
Expansion in Prompt Search
Applying Classic Optimization Techniques to Prompt Optimization
A team is developing a system to automatically find the best prompt for summarizing legal documents. Their process is as follows:
- They create a large, diverse list of 100 potential prompts.
- They use a small, representative dataset to calculate an accuracy score for each of the 100 prompts.
- They select the prompt with the highest accuracy score from the initial list and the process concludes.
Which critical element of an effective search strategy is missing from their approach?
Evaluating Prompt Search Strategies
Critique of a Prompt Finding Method