Learn Before
Evaluating Prompt Search Strategies
A startup is optimizing a prompt for a chatbot that handles a single, well-defined task: processing product returns. They have a limited budget for computation and need to find a good prompt quickly. They are considering two different search strategies:
- Strategy A: Systematically generate and test 10,000 distinct prompt variations against a validation dataset, then select the single highest-performing prompt.
- Strategy B: Start with 5 initial prompts. In each step, test the current set, discard the poor performers, and generate a few new variations based on the most successful ones. Repeat this process until the performance score stops improving.
Based on the startup's constraints, which strategy would you recommend? Justify your choice by evaluating the primary trade-off between these two approaches.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Iterative LLM-Based Prompt Search
Expansion in Prompt Search
Applying Classic Optimization Techniques to Prompt Optimization
A team is developing a system to automatically find the best prompt for summarizing legal documents. Their process is as follows:
- They create a large, diverse list of 100 potential prompts.
- They use a small, representative dataset to calculate an accuracy score for each of the 100 prompts.
- They select the prompt with the highest accuracy score from the initial list and the process concludes.
Which critical element of an effective search strategy is missing from their approach?
Evaluating Prompt Search Strategies
Critique of a Prompt Finding Method