Learn Before
Evaluating a Prompt Optimization Strategy
Based on the following scenario, evaluate the soundness of the engineer's proposed plan for finding the best system instruction. Justify your evaluation by referencing the inherent difficulties of this optimization process.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A research team is trying to find the most effective instruction to guide a large language model in summarizing legal documents. They first try asking the model to rate a list of 100 different candidate instructions on a scale of 1-10 for 'clarity and effectiveness'. They then discover that the model's ratings do not correlate well with which instructions actually produce the best summaries when tested. Furthermore, the process of generating and evaluating summaries for all 100 instructions is taking several days and consuming their entire computation budget. Which statement best analyzes the fundamental difficulties the team is facing?
Evaluating a Prompt Optimization Strategy
Critique of a Prompt Optimization Method
The most efficient method for optimizing a prompt's instruction for a new task is to have the large language model (LLM) score a list of potential instructions for quality, and then select the highest-scoring one for use, thereby avoiding the high computational cost of testing each instruction on the actual task.