Learn Before
Critique of a Prompt Optimization Method
A developer proposes a method to quickly find the best instruction for a language model: have the model generate 1,000 different instructions and then internally rank them from best to worst, selecting the top-ranked one. This is intended to avoid the costly process of testing each instruction on the actual task. Based on the known difficulties in learning optimal instructions, identify and explain the primary flaw in this proposed method.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A research team is trying to find the most effective instruction to guide a large language model in summarizing legal documents. They first try asking the model to rate a list of 100 different candidate instructions on a scale of 1-10 for 'clarity and effectiveness'. They then discover that the model's ratings do not correlate well with which instructions actually produce the best summaries when tested. Furthermore, the process of generating and evaluating summaries for all 100 instructions is taking several days and consuming their entire computation budget. Which statement best analyzes the fundamental difficulties the team is facing?
Evaluating a Prompt Optimization Strategy
Critique of a Prompt Optimization Method
The most efficient method for optimizing a prompt's instruction for a new task is to have the large language model (LLM) score a list of potential instructions for quality, and then select the highest-scoring one for use, thereby avoiding the high computational cost of testing each instruction on the actual task.